Skip to content

Command Line Tools

The MAX-ARD SDK CLI tools have built-in capabilities for working with ARD data stored in an Amazon S3 location. For example, the max-ard storage describe command describes the data in the specified S3 bucket. This command can also be used to generate a QGIS layer definition of the files stored in S3.

See example requests that use max-ard storage describe.

In addition to the Maxar ARD SDK CLI tools, the following tools are useful for working with ARD imagery in cloud storage:

AWS Command Line Interface

Amazon's AWS CLI is a set of command line tools for interacting with AWS services. The AWS CLI can be used to access and work with ARD data stored in an S3 bucket.

Installation

To install AWS CLI, see the AWS CLI Installation Guide. Choose your operation system from the list.

Once you've installed the AWS CLI, you'll need to configure it to use your AWS credentials to access data in your S3 bucket. See the AWS CLI Configuration Guide for details.

Accessing the Maxar ARD public datasets

The datasets used in the examples below are publicly available and do not require authentication. However, AWS CLI still checks for a current token for access. If you get an ExpiredToken error while running any of the AWS CLI examples below, you can append --no-sign-request to the command to skip the check.

Example:

aws s3 ls maxar-ard-samples/sample-002/ --no-sign-request

A sample dataset over San Francisco can be downloaded from the Maxar website. You will be asked to fill out a contact form before accessing the data. Download Sample Data

Use AWS CLI commands to work with ARD data

aws s3 ls (list files)

The ls command lists the files and prefixes in the requested path. In this example, we want to list the contents of the maxar-ard-samples/sample-002/ location.

aws s3 ls maxar-ard-samples/sample-002/

PRE 13/
PRE acquisition_collections/
PRE order_collections/
2021-04-12 11:11:02      243 5633324896324598239-request.json
The results show that there are three prefixes or subdirectories and a .json file in the sample-002

Note: Listing the contents of a prefix requires the trailing slash / at the end of the prefix name. If the trailing slash is not included, the results will only show the name of the prefix in the request. In this case, it will show sample-002.

To list all files within the sample-002 prefix and its descendants, include --recursive in your request.

aws s3 ls maxar-ard-samples/sample-002/ --recursive

2021-04-12 19:02:12      98304 sample-002/13/031133320010/2020-07-29/10300100AA1C6800-cloud-mask.gpkg
2021-04-12 19:02:12      98304 sample-002/13/031133320010/2020-07-29/10300100AA1C6800-data-mask.gpkg
2021-04-12 19:02:20   71193652 sample-002/13/031133320010/2020-07-29/10300100AA1C6800-ms.tif
2021-04-12 19:02:15  135381264 sample-002/13/031133320010/2020-07-29/10300100AA1C6800-pan.tif
2021-04-12 19:02:12   61112300 sample-002/13/031133320010/2020-07-29/10300100AA1C6800-visual.tif
2021-04-12 19:02:23       2104 sample-002/13/031133320010/2020-07-29/10300100AA1C6800.json
2021-04-12 18:37:40      98304 sample-002/13/031133320010/2020-07-29/10300100AC94D700-cloud-mask.gpkg
2021-04-12 18:37:40      98304 sample-002/13/031133320010/2020-07-29/10300100AC94D700-data-mask.gpkg
2021-04-12 18:37:49   74875710 sample-002/13/031133320010/2020-07-29/10300100AC94D700-ms.tif
2021-04-12 18:37:43  142014484 sample-002/13/031133320010/2020-07-29/10300100AC94D700-pan.tif
2021-04-12 18:37:40   61868775 sample-002/13/031133320010/2020-07-29/10300100AC94D700-visual.tif
2021-04-12 18:37:51       2106 sample-002/13/031133320010/2020-07-29/10300100AC94D700.json
2021-04-12 18:36:36       1648 sample-002/acquisition_collections/10300100AC700900_collection.json
2021-04-12 18:38:09       4122 sample-002/acquisition_collections/10300100AC94D700_collection.json
2021-04-12 20:21:33       1078 sample-002/acquisition_collections/10300100ACCDAB00_collection.json
2021-04-12 18:37:44       1650 sample-002/acquisition_collections/10300100AD437400_collection.json
2021-04-12 15:13:26       2982 sample-002/acquisition_collections/10300100B2B49700_collection.json
2021-04-12 15:13:02       1648 sample-002/acquisition_collections/10300100B3841C00_collection.json
2021-04-12 15:13:27       2597 sample-002/acquisition_collections/10300100B39ACD00_collection.json
2021-02-26 14:40:48       2767 sample-002/order_collections/5600806342510769702_root_collection.json
2021-04-12 20:37:37       2749 sample-002/order_collections/5633324896324598239_root_collection.json
[results truncated]

Partial match filtering

Partial match filtering is performed automatically when an AWS CLI request is made. This is helpful when you only know part of the string, or when you want to find all files that match the partial path.

This example:

aws s3 ls maxar-ard-samples/sample-002/acquisition_collections/10300100
returns all files in the acquisition_collections prefix that start with 10300100.

2021-04-12 20:19:36        699 10300100A67D3100_collection.json
2021-04-12 20:18:47       1078 10300100A9547600_collection.json
2021-04-12 20:37:33       1268 10300100A9CC9200_collection.json
2021-04-12 19:02:32       3738 10300100AA1C6800_collection.json
2021-04-12 19:11:34       1647 10300100AB101A00_collection.json
2021-04-12 18:36:36       1648 10300100AC700900_collection.json
2021-04-12 18:38:09       4122 10300100AC94D700_collection.json
2021-04-12 20:21:33       1078 10300100ACCDAB00_collection.json
2021-04-12 18:37:44       1650 10300100AD437400_collection.json
2021-04-12 15:13:26       2982 10300100B2B49700_collection.json
2021-04-12 15:13:02       1648 10300100B3841C00_collection.json
2021-04-12 15:13:27       2597 10300100B39ACD00_collection.json

Find files that include a specific string

Use grep (linux, macOS) or findstr (windows) to find all of the files that include a specific string in the path.

Example: Find all files named visual.tif in the rio_deforestation dataset.

aws s3 ls maxar-ard-samples/rio_deforestation/ --recursive | grep "visual.tif"

2021-04-12 20:10:25   56654384 rio_deforestation/19/300022033202/2009-09-05/105041000218BB00-visual.tif
2021-04-12 20:10:15   28992496 rio_deforestation/19/300022033202/2010-07-01/1050410001B2E100-visual.tif
2021-04-12 18:48:08   56860968 rio_deforestation/19/300022033202/2011-05-11/103001000AC5B000-visual.tif
2021-04-12 19:21:27   66317737 rio_deforestation/19/300022033202/2011-08-12/103001000CB3FD00-visual.tif
2021-04-12 19:06:27   30197512 rio_deforestation/19/300022033202/2012-05-10/103001001826FC00-visual.tif
2021-04-12 17:57:07   29645545 rio_deforestation/19/300022033202/2012-09-04/1050410000818900-visual.tif
2021-04-12 19:10:00   58833258 rio_deforestation/19/300022033202/2013-05-09/1050410002473100-visual.tif
2021-04-12 19:09:01   55904842 rio_deforestation/19/300022033202/2013-08-17/10300100253F7900-visual.tif
2021-04-12 19:06:25   31344763 rio_deforestation/19/300022033202/2013-08-28/1030010026C30800-visual.tif
2021-04-12 19:11:53   34979004 rio_deforestation/19/300022033202/2013-12-23/10300100296ED400-visual.tif
2021-04-12 17:58:54   15285643 rio_deforestation/19/300022033202/2014-04-21/10504100105F9E00-visual.tif
2021-04-12 19:05:07   61825502 rio_deforestation/19/300022033202/2015-06-29/1030010044098C00-visual.tif
[truncated]

Example: List all files with acquisition ID "1050410000818900" in the path.

aws s3 ls maxar-ard-samples/rio_deforestation/ --recursive | grep "1050410000818900"

2021-04-12 17:57:07      98304 rio_deforestation/19/300022033202/2012-09-04/1050410000818900-cloud-mask.gpkg
2021-04-12 17:57:07      98304 rio_deforestation/19/300022033202/2012-09-04/1050410000818900-data-mask.gpkg
2021-04-12 17:57:14   35145697 rio_deforestation/19/300022033202/2012-09-04/1050410000818900-ms.tif
2021-04-12 17:57:09  128922686 rio_deforestation/19/300022033202/2012-09-04/1050410000818900-pan.tif
2021-04-12 17:57:07   29645545 rio_deforestation/19/300022033202/2012-09-04/1050410000818900-visual.tif
2021-04-12 17:57:15       2390 rio_deforestation/19/300022033202/2012-09-04/1050410000818900.json
2021-04-12 17:57:19        698 rio_deforestation/acquisition_collections/1050410000818900_collection.json

aws s3 cp (copy files)

The cp command copies files from s3 to a specified location. These examples show copying from the ARD sample data in S3 to a local directory.

Copy a specific file

This example copies the 1050410000818900.json file from its s3 location to the user's current local directory.

aws s3 cp s3://maxar-ard-samples/rio_deforestation/19/300022033202/2012-09-04/1050410000818900.json ./1050410000818900.json
The response below indicates the file was downloaded.

download: s3://maxar-ard-samples/rio_deforestation/19/300022033202/2012-09-04/1050410000818900.json to ./1050410000818900.json

Copy all files in a prefix

This example shows copying all the files from the specified prefix to the user's current local directory using the --recursive option.

aws s3 cp s3://maxar-ard-samples/rio_deforestation/19/300022033202/2012-09-04/ ./ --recursive

download: s3://maxar-ard-samples/rio_deforestation/19/300022033202/2012-09-04/1050410000818900.json to ./1050410000818900.json
download: s3://maxar-ard-samples/rio_deforestation/19/300022033202/2012-09-04/1050410000818900-cloud-mask.gpkg to ./1050410000818900-cloud-mask.gpkg
download: s3://maxar-ard-samples/rio_deforestation/19/300022033202/2012-09-04/1050410000818900-data-mask.gpkg to ./1050410000818900-data-mask.gpkg
download: s3://maxar-ard-samples/rio_deforestation/19/300022033202/2012-09-04/1050410000818900-ms.tif to ./1050410000818900-ms.tif
download: s3://maxar-ard-samples/rio_deforestation/19/300022033202/2012-09-04/1050410000818900-visual.tif to ./1050410000818900-visual.tif
download: s3://maxar-ard-samples/rio_deforestation/19/300022033202/2012-09-04/1050410000818900-pan.tif to ./1050410000818900-pan.tif

aws s3 sync (synchronize files in two locations)

Use the sync command to synchronize the contents of two locations. This is typically used to keep files up to date. In this example, rio-deforestation is the source and rio-data is the target local directory.

aws s3 sync s3://maxar-ard-samples/rio_deforestation rio-data

Completed 2.1 KiB/~7.1 GiB (5.7 KiB/s) with ~146 file(s) remaining (calculating.download: s3://maxar-ard-samples/rio_deforestation/19/300022033202/2009-09-05/105041000218BB00.json to rio-data/19/300022033202/2009-09-05/105041000218BB00.json
Completed 2.1 KiB/~7.1 GiB (5.7 KiB/s) with ~145 file(s) remaining (calculating.Completed 98.1 KiB/~7.1 GiB (194.9 KiB/s) with ~146 file(s) remaining (calculatidownload: s3://maxar-ard-samples/rio_deforestation/19/300022033202/2009-09-05/105041000218BB00-cloud-mask.gpkg to rio-data/19/300022033202/2009-09-05/105041000218BB00-cloud-mask.gpkg
  [truncated]

GDAL/OGR command line tools

Introduction

Files stored in S3 can be accessed using GDAL/OGR command line tools. Use gdalinfo to access raster data and ogrinfo to access vector data. For example, use gdalinfo to list information about a GeoTIFF, and 'ogrinfo' to list information about a cloud mask GeoPackage. Examples of both of these are shown below.

To access files stored in S3 using GDAl or OGR, use vsis3.

Installation

The GDAL/OGR CLI installs with GDAL. To find the latest release and download instructions, see GDAL Current Releases.

Use GDAL command line tools to work with ARD data in the cloud or in a local directory.

GDAL's vsis3 virtual driver can read files from S3. To use it, change the s3:// protocol in the file's URL to `/vsis3/:

Protocol File URL
S3 s3://maxar-ard-samples/rio_deforestation/19/300022033202/2013-08-28/1030010026C30800-ms.tif
GDAL /vsis3/maxar-ard-samples/rio_deforestation/19/300022033202/2013-08-28/1030010026C30800-ms.tif
local file /[local dir/subdir]/19/300022033202/2013-08-28/1030010026C30800-ms.tif

gdalinfo

This example lists information about the specified GeoTIFF (raster file).

gdalinfo /vsis3/maxar-ard-samples/rio_deforestation/19/300022033202/2013-08-28/1030010026C30800-ms.tif

Example using downloaded files from the sample dataset (SF)

gdalinfo /[root dir]/[subdir]/sample-001-subset/10/120020223032/2019-07-12/1050010017603F00-ms.tif

Driver: GTiff/GeoTIFF
Files: /vsis3/maxar-ard-samples/rio_deforestation/19/300022033202/2013-08-28/1030010026C30800-ms.tif
Size is 2383, 2383
Coordinate System is:
PROJCRS["WGS 84 / UTM zone 19S",
    BASEGEOGCRS["WGS 84",
        DATUM["World Geodetic System 1984",
            ELLIPSOID["WGS 84",6378137,298.257223563,
                LENGTHUNIT["metre",1]]],
        PRIMEM["Greenwich",0,
            ANGLEUNIT["degree",0.0174532925199433]],
        ID["EPSG",4326]],
    CONVERSION["UTM zone 19S",
        METHOD["Transverse Mercator",
            ID["EPSG",9807]],
        PARAMETER["Latitude of natural origin",0,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8801]],
        PARAMETER["Longitude of natural origin",-69,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8802]],
        PARAMETER["Scale factor at natural origin",0.9996,
            SCALEUNIT["unity",1],
            ID["EPSG",8805]],
        PARAMETER["False easting",500000,
            LENGTHUNIT["metre",1],
            ID["EPSG",8806]],
        PARAMETER["False northing",10000000,
            LENGTHUNIT["metre",1],
            ID["EPSG",8807]]],
    CS[Cartesian,2],
        AXIS["(E)",east,
            ORDER[1],
            LENGTHUNIT["metre",1]],
        AXIS["(N)",north,
            ORDER[2],
            LENGTHUNIT["metre",1]],
    USAGE[
        SCOPE["Engineering survey, topographic mapping."],
        AREA["Between 72°W and 66°W, southern hemisphere between 80°S and equator, onshore and offshore. Argentina. Bolivia. Brazil. Chile. Colombia. Peru."],
        BBOX[-80,-72,0,-66]],
    ID["EPSG",32719]]
Data axis to CRS axis mapping: 1,2
Origin = (619843.750000000000000,8895156.250000000000000)
Pixel Size = (2.229332773814519,-2.229332773814519)
Metadata:
  AREA_OR_POINT=Area
  TIFFTAG_COPYRIGHT=(C) Copyright 2021 DigitalGlobe, Inc., Westminster CO USA 80234
Image Structure Metadata:
  COMPRESSION=DEFLATE
  INTERLEAVE=PIXEL
  LAYOUT=COG
Corner Coordinates:
Upper Left  (  619843.750, 8895156.250) ( 67d54'23.67"W,  9d59'35.02"S)
Lower Left  (  619843.750, 8889843.750) ( 67d54'23.10"W, 10d 2'27.97"S)
Upper Right (  625156.250, 8895156.250) ( 67d51'29.21"W,  9d59'34.43"S)
Lower Right (  625156.250, 8889843.750) ( 67d51'28.60"W, 10d 2'27.38"S)
Center      (  622500.000, 8892500.000) ( 67d52'56.14"W, 10d 1' 1.20"S)
Band 1 Block=512x512 Type=UInt16, ColorInterp=Gray
  NoData Value=0
  Overviews: 1192x1192, 596x596, 298x298
Band 2 Block=512x512 Type=UInt16, ColorInterp=Undefined
  NoData Value=0
  Overviews: 1192x1192, 596x596, 298x298
Band 3 Block=512x512 Type=UInt16, ColorInterp=Undefined
  NoData Value=0
  Overviews: 1192x1192, 596x596, 298x298
Band 4 Block=512x512 Type=UInt16, ColorInterp=Undefined
  NoData Value=0
  Overviews: 1192x1192, 596x596, 298x298
Band 5 Block=512x512 Type=UInt16, ColorInterp=Undefined
  NoData Value=0
  Overviews: 1192x1192, 596x596, 298x298
Band 6 Block=512x512 Type=UInt16, ColorInterp=Undefined
  NoData Value=0
  Overviews: 1192x1192, 596x596, 298x298
Band 7 Block=512x512 Type=UInt16, ColorInterp=Undefined
  NoData Value=0
  Overviews: 1192x1192, 596x596, 298x298
Band 8 Block=512x512 Type=UInt16, ColorInterp=Undefined
  NoData Value=0
  Overviews: 1192x1192, 596x596, 298x298

ogrinfo

This example lists information about a specified cloud mask GeoPackage (vector file).

ogrinfo /vsis3/maxar-ard-samples/rio_deforestation/19/300022033202/2013-08-28/1030010026C30800-cloud-mask.gpkg

Example using file from downloaded sample data:

ogrinfo /[root dir]/[subdir]sample-001-subset/10/120020223032/2019-07-12/1050010017603F00-cloud-mask.gpkg

INFO: Open of `/vsis3/maxar-ard-samples/rio_deforestation/19/300022033202/2013-08-28/1030010026C30800-cloud-mask.gpkg'
      using driver `GPKG' successful.
1: 19_300022033202_1030010026C30800-cloud-mask (Multi Polygon)

Additional GDAL commands are described in the GDAL programs documentation.

Rasterio's "rio" command line tools

Rio is the Rasterio CLI. Use rio commands to get information about a file or to visualize a file.

Installation

Basic installation instructions for Rasterio can be found on the Rasterio Documentation Site.

Use rio to work with ARD Data

Once you install rasterio, you can use rio commands to access and work with ARD data. If the rio-viz subpackage is not installed, follow these instructions.

rio uses the same connection syntax as GDAL, so s3:// URLs should be changed to `/vsis3/.

rio info

rio info /vsis3/maxar-ard-samples/rio_deforestation/19/300022033202/2013-08-28/1030010026C30800-ms.tif

Example using downloaded sample data:

{"blockxsize": 512, "blockysize": 512, "bounds": [619843.75, 8889843.75, 625156.25, 8895156.25], "colorinterp": ["gray", "undefined", "undefined", "undefined", "undefined", "undefined", "undefined", "undefined"], "compress": "deflate", "count": 8, "crs": "EPSG:32719", "descriptions": [null, null, null, null, null, null, null, null], "driver": "GTiff", "dtype": "uint16", "height": 2383, "indexes": [1, 2, 3, 4, 5, 6, 7, 8], "interleave": "pixel", "lnglat": [-67.88226237156347, -10.017000640120592], "mask_flags": [["nodata"], ["nodata"], ["nodata"], ["nodata"], ["nodata"], ["nodata"], ["nodata"], ["nodata"]], "nodata": 0.0, "res": [2.2293327738145194, 2.2293327738145194], "shape": [2383, 2383], "tiled": true, "transform": [2.2293327738145194, 0.0, 619843.75, 0.0, -2.2293327738145194, 8895156.25, 0.0, 0.0, 1.0], "units": [null, null, null, null, null, null, null, null], "width": 2383}

rio viz

The rio viz command line tool provides a visual browser to explore ARD image tiles.

rio viz /vsis3/maxar-ard-samples/sample-002/13/031133231113/2021-01-16/10300100B3841C00-visual.tif
Visualization Example

Example using the downloaded sample data:

rio viz /[root dir]/[subdir]/sample-001-subset/10/120020223032/2019-07-12/1050010017603F00-visual.tif
Download Visualization Example

Rio viz can also read the ARD STAC Items.

Example:

rio viz https://maxar-ard-samples.s3.amazonaws.com/sample-002/13/031133231113/2021-01-16/10300100B3841C00.json --reader rio_tiler.io.STACReader
Back to top