Recent measurements

POST {baseUrl}/v2/recent-datasource-measurements-query

By Device

This query is only by Datasource ID. To query by device or reference site, use the historical measurements report.

Returns a list of recent measurements in time descending order.

Example
url: https://clarity-data-api.clarity.io/v2/recent-datasource-measurements-query
body:

{
    "org": "myorg1234",
    "datasourceIds": ["DS123456", "DS987654"],
    "outputFrequency": "hour"
}

Cached measurements

Recent measurements are served from a fast cache that goes back in time depending on output frequency:

Individual measurements (minute) cached for 24 hours
Hourly aggregations (hour) cached for 48 hours
Daily aggregations (day) cached for 10 days

Continuations

A common access pattern is to routinely poll to get any new measurements since the last time you asked. It is awkward to accomplish this using startTime on your requests. Due to race conditions it is possible that you could miss a measurement, or get the same measurement twice. To make this easier, a companion endpoint accepts continuation tokens.

In your first call — using this endpoint — you specify an additional parameter: replyWithContinuationToken: true. The response will include an x-clarity-continuation-token in the header.

On subsequent calls, use the recent datasource measurements continuation endpoint and pass that continuation token back in the body (along with org). The continuation response will pick up from the last call, returning newer measurements (if any exist) for the same datasources requested in the first call. An updated continuation token will be returned as well.

You can continue making continuation queries, with the token being updated on each call. A continuation token is good for 24 hours after it is issued.

Note: If the original call was for allDatasources in the org, then the continuations will include/exclude datasources as they are subscribed/unsubscribed.

Headers

parameter		description
`x-api-key`	string	The API key string.
`Accept-Encoding`	string	Encoding options, `gzip` is supported to compress data. Note: We strongly recommend including it in the headers to reduce response payload size. Example: `Accept-Encoding: gzip`

Request

You must include org on all requests. Provide a list of datasources in datasourceIds, or for all the organization's datasources, pass "allDatasources": true.

body parameter		description
`org`	string required	Find the org ID by clicking on your user icon in Dashboard. Example org ID: `"myorgVD43"`
`allDatasources`	bool	• `true` to retrieve measurements for all datasources in the specified organization for which you have access. • `false` or omit the parameter if you want to retrict using `datasourceIds`.
`datasourceIds`	array of string	A list of modern datasource ids as returned from `/v2/datasources`. Only measurements from periods when the datasource was subscribed by the requested org are returned.
`replyWithContinuationToken`	bool	`true` if you want to receive a continuation token in the response headers, otherwise omit this parameter (see Continuations above)

And then add these parameters

additional query parameter		description
`outputFrequency`	string	The output frequency of the aggregations to return in the measurement. One of: • `"minute"` default - returns individual measurements, with no aggregation applied. • `"hour"` returns 1-Hour Mean, 24-Hour Rolling Mean and NowCast aggregations, and AQIs which are calculated on these aggregations as per their standard. • `"day"` returns 24-Hour Mean aggregations, and AQIs which are calculated on these aggregations as per their standard.
`format`	string	The format of the response. One of: • `json-long` default - A JSON object per metric in a measurement • `csv-wide` CSV, one row per measurement
`startTime`	string	Cut-off time for the earliest measurement desired. If not specified, the start time used will be chosen based on the requested output frequency `minute` → 1 hour prior to time of request `hour` → 24 hours prior to time of request `day` → 48 hours prior to time of request Measurements are returned from a cache. Earlier times are accepted in the parameter, but only data from the cache will be returned. Example: `startTime: "2023-09-01T00:00:00Z"`
`metricSelect`	string	Limits the metrics returned in the measurement. See Metrics selection
`qcAssessment`	boolean	When `true`, adds the field `qcAssessment` to each metric in the response. Default `false`. Values are `valid` and `invalid` but may be expanded in the future.
`qcFlags`	boolean	When `true`, adds the field `qcFlags` to each metric in the response. Default `false`. See QC Flags

Response Headers

parameter		description
`x-clarity-continuation-token`	string	The continuation token, if it was requested with `replyWithContinuationToken`

Response Formats

json-long

A typical JSON structure including these important objects

request - an echo back of the original request
data - a list of metrics objects
locations - a sidebar dictionary to lookup the location of a datasource

The time field for hour or day aggregate measurements is the start of the period for the aggregate. This matches how UIs typically label hour or day aggregated information.

This example is an excerpt of a much longer response:

    "request": {
        "format": "json-long",
        "org": "myorg1234",
        "outputFrequency": "hour",
        "allDatasources": true,
        "startTime": "2023-11-03T19:00:00+00:00"
        "qcAssessment": true,
        "qcFlags": true,
    },

    "data": [
        {
            "datasourceId": "DCQYT1459"
            "time": "2023-11-03T19:00:00.000Z",
            "status": "sensor-ready",
            "value": 20.25,
            "raw": 20.25,
            "metric": "temperatureInternal1HourMean",
            "qcAssessment": "valid",
            "qcFlags": []
        },
        {
            "datasourceId": "DCQYT1459"
            "time": "2023-11-03T19:00:00.000Z",
            "status": "calibrated-ready",
            "value": 8192.31,
            "raw": 3019.89,
            "metric": "pm2_5ConcMass1HourMean",
            "qcAssessment": "invalid",
            "qcFlags": ["QC.I.OOB.001", "QC.I.OOB.002"]
        },
        ...more...
    ],

    "locations": [
        {
            "datasourceId": "DCQYT1459",
            "lon": -122.30173,
            "lat": 37.87864
        },
        {
            "datasourceId": "DDDYN3547",
            "lon": -116.7845,
            "lat": 38.28068
        },
        ...more...
    ]

Note: Best practice is that if a device in a datasource is replaced, the new device should be configured to the same location as the old device. If that does not happen, then it is possible for location to change in the middle of a time-series. If that should occur, then the locations object above will show both locations and identify the first measurement in the time-series that has the new location:

        {
            "datasourceId": "DABCD1234",
            "lon": -120.24056,
            "lat": 37.87864
        },
        {
            "datasourceId": "DABCD1234",               <--- same datasource
            "lon": -122.00052,                         <--- new location
            "lat": 38.28068,
            "changedAt": "2023-11-03T21:00:00.000Z"    <--- first measurement at new location
        },

csv-wide

Column Headers

Automation that processes this response should not assume specifc columns or column ordering. Interpret based on the header row of the response.

The response is like a CSV file: A comma-separated string with an initial header row identifying the columns and with newlines separating the measurements. The set of columns returned depends on the outputFrequency requested.

Identifying columns for all output frequencies:

column header	description
datasourceId	Unique identifier of the datasource. (string)
sourceId	Unique identifier of the underlying device or reference site. (string)
sourceType	Either `CLARITY_NODE` or `REFERENCE_SITE`. (string)
outputFrequency	The output frequency; one of `minute`, `hour`, or `day`. (string)

Example with many metrics selected:

datasourceId,sourceId,sourceType,periodType,startOfPeriod,endOfPeriod,locationLatitude,locationLongitude,atmPressureRaw1HourMean,no2ConcCalibrated1HourMean,no2ConcCalibrated1HourMeanUsEpaAqi,no2ConcCalibratedWaDwerAqi,no2ConcRaw1HourMean,no2ConcRaw1HourMeanUsEpaAqi,no2ConcRawWaDwerAqi,o3ConcRaw1HourMean,o3ConcRawWaDwerAqi,pm10ConcMassCalibrated1HourMean,pm10ConcMassCalibrated24HourRollingMean,pm10ConcMassCalibratedWaDwerAqi,pm10ConcMassRaw1HourMean,pm10ConcMassRaw24HourRollingMean,pm10ConcMassRawWaDwerAqi,pm10ConcNumRaw1HourMean,pm10ConcNumRaw24HourRollingMean,pm1ConcMassCalibrated1HourMean,pm1ConcMassRaw1HourMean,pm1ConcNumRaw1HourMean,pm2_5ConcMassCalibrated1HourMean,pm2_5ConcMassCalibrated24HourRollingMean,pm2_5ConcMassCalibratedNowCast,pm2_5ConcMassCalibratedNowCastAqi,pm2_5ConcMassCalibratedWaDwerAqi,pm2_5ConcMassRaw1HourMean,pm2_5ConcMassRaw24HourRollingMean,pm2_5ConcMassRawNowCast,pm2_5ConcMassRawNowCastAqi,pm2_5ConcMassRawWaDwerAqi,pm2_5ConcNumRaw1HourMean,pm2_5ConcNumRaw24HourRollingMean,relHumidAmbientRaw1HourMean,relHumidInternalRaw1HourMean,temperatureAmbientRaw1HourMean,temperatureInternalRaw1HourMean,windDirectionRaw1HourMean,windSpeedRaw1HourMean
DCQYT1459,AXGCR7WJ,CLARITY_NODE,hour,2023-09-01T17:00:00Z,2023-09-01T18:00:00Z,37.878635,-122.301728,,,,,,,,,,,,,0.71,12.45,1,0.91,13.27,,0.0,0.85,,,,,,0.0,9.67,0.02,0,0,0.88,13.18,,54.68,,21.33,,
DDWLW0383,SF3FLYVL,CLARITY_NODE,hour,2023-09-01T17:00:00Z,2023-09-01T18:00:00Z,19.746,-155.087,,,,,9.23,9,9,,,,,,7.27,7.27,7,8.76,8.76,,4.69,8.38,,,,,,7.08,7.08,7.08,30,7,8.72,8.72,,45.12,,21.81,,
DEFSL2364,RW1075XQ,REFERENCE_SITE,hour,2023-09-01T17:00:00Z,2023-09-01T18:00:00Z,45.4369,-75.7261,,,,,,,,31.0,31,,,,,,,,,,,,6.4,,,,6,6.4,,,,6,,,,,,,,
...more...

Example with qcAssessment and qcFlags:

"datasourceId","sourceId","sourceType","outputFrequency","startOfPeriod","endOfPeriod","locationLatitude","locationLongitude","pm2_5ConcMass1HourMean.qcAssessment","pm2_5ConcMass1HourMean.qcFlags","pm2_5ConcMass1HourMean.raw","pm2_5ConcMass1HourMean.status","pm2_5ConcMass1HourMean.value"
DNIXT8160,AVWPGWWW,CLARITY_NODE,hour,2024-10-07T17:00:00Z,2024-10-07T18:00:00Z,37.87902695401437,-122.30192899608234,invalid,"[""QC.I.OOB.001"",""QC.I.OOB.002""]",5896.15,calibrated-ready,25.75

Example code

Slicing specific columns from CSV

The following sample Python code selects just the columns you want and converts to native Python types.

# simple demo using Clarity Data API

import requests
import os
import csv
import pprint
import datetime
import json


BASEURL = 'https://clarity-data-api.clarity.io'
HEADERS = {
    'Accept-Encoding': 'gzip',
    'x-api-key': os.environ.get('MY_CLARITY_API_KEY') # put your key in the environment or directly here
}
ORG_ID = '<your org id goes here>'  # you can find it in Dashboard under the menu item Organization>Resources
DATASOURCE_ID = '<the id of a datasource in your org>'


def check_can_connect():
    # verify can reach the API
    response = requests.get(BASEURL, HEADERS)
    http_code = response.status_code
    connected = http_code == 200
    if connected:
        print("Connected to Clarity")
    else:
        print(f"{http_code}  :(  Cannot connect")


def get_recent_measurements(
    org, datasource_ids, output_frequency, format, metric_selector
):
    # Fetch measurements from the API
    url = BASEURL + "/v2/recent-datasource-measurements-query"
    request_body = {
        "org": org,
        "datasourceIds": datasource_ids,
        "outputFrequency": output_frequency,
        "format": format,
        "metricSelect": metric_selector,
        "qcAssessment": True,
        "qcFlags": True,
    }
    response = requests.post(url, headers=HEADERS, json=request_body)
    response.raise_for_status()
    return response.text


def csv_to_typed(csv_data, fields_to_extract):
    # slice specific columns out of the CSV-style response
    # convert each to best Python type
    # return dictionary

    def convert_to_best_type(value, name):
        # Try these types in order: float, datetime, else string
        if value and ("qcFlags" in name or "Array" in name or "Vector" in name):
            try:
                return json.loads(value)
            except:
                pass
        try:
            return float(value)
        except Exception:
            try:
                return datetime.datetime.strptime(value, "%Y-%m-%dT%H:%M:%SZ")
            except Exception:
                return value

    reader = csv.DictReader(csv_data.splitlines())
    return [
        {field: convert_to_best_type(row[field], field) for field in fields_to_extract}
        for row in reader
    ]


check_can_connect()

tabular = get_recent_measurements(
    org=ORG_ID,
    datasource_ids=[DATASOURCE_ID],
    format="csv-wide",
    output_frequency="hour",
    metric_selector="only pm2_5ConcMass1HourMean",
)

measurements = csv_to_typed(
    tabular,
    [
        "startOfPeriod",
        "pm2_5ConcMass1HourMean.value",
        "pm2_5ConcMass1HourMean.status",
        "pm2_5ConcMass1HourMean.qcAssessment",
        "pm2_5ConcMass1HourMean.qcFlags",
    ],
)

pprint.pprint(measurements)