Python library to work with CSVW described tabular data
This package provides
This package runs under Python >=3.8, use pip to install:
$ pip install csvw
csvw2json
Converting CSVW data to JSON
$ csvw2json tests/fixtures/zipped-metadata.json
{
"tables": [
{
"url": "tests/fixtures/zipped.csv",
"row": [
{
"url": "tests/fixtures/zipped.csv#row=2",
"rownum": 1,
"describes": [
{
"ID": "abc",
"Value": "the value"
}
]
},
{
"url": "tests/fixtures/zipped.csv#row=3",
"rownum": 2,
"describes": [
{
"ID": "cde",
"Value": "another one"
}
]
}
]
}
]
}
csvwvalidate
Validating CSVW data
$ csvwvalidate tests/fixtures/zipped-metadata.json
OK
csvwdescribe
Describing tabular-data files with CSVW metadata
$ csvwdescribe --delimiter "|" tests/fixtures/frictionless-data.csv
{
"@context": "http://www.w3.org/ns/csvw",
"dc:conformsTo": "data-package",
"tables": [
{
"dialect": {
"delimiter": "|"
},
"tableSchema": {
"columns": [
{
"datatype": "string",
"name": "FK"
},
{
"datatype": "integer",
"name": "Year"
},
{
"datatype": "string",
"name": "Location name"
},
{
"datatype": "string",
"name": "Value"
},
{
"datatype": "string",
"name": "binary"
},
{
"datatype": "string",
"name": "anyURI"
},
{
"datatype": "string",
"name": "email"
},
{
"datatype": "string",
"name": "boolean"
},
{
"datatype": {
"dc:format": "application/json",
"base": "json"
},
"name": "array"
},
{
"datatype": {
"dc:format": "application/json",
"base": "json"
},
"name": "geojson"
}
]
},
"url": "tests/fixtures/frictionless-data.csv"
}
]
}
Find the Python API documentation at csvw.readthedocs.io.
A quick example for using csvw
from Python code:
import json
from csvw import CSVW
data = CSVW('https://raw.githubusercontent.com/cldf/csvw/master/tests/fixtures/test.tsv')
print(json.dumps(data.to_json(minimal=True), indent=4))
[
{
"province": "Hello",
"territory": "world",
"precinct": "1"
}
]
utf-8-sig
codecs.
Thus, if such data starts with U+FEFF
this will be interpreted as BOM
and skipped.csv
module in Python's standard library. Thus, if a commentPrefix
is specified in a Dialect
instance, this will lead to skipping rows where the first value starts
with commentPrefix
, even if the value was quoted.escapechar
may not be round-tripped as expected (when specifying
escapechar
or a csvw.Dialect
with quoteChar
but doubleQuote==False
),
when minimal quoting is specified. This is due to inconsistent csv
behaviour
across Python versions (see https://bugs.python.org/issue44861).While we use the CSVW specification as guideline, this package does not (and probably never will) implement the full extent of this spec.
tableSchema
, but instead are matched based on the
CSV column header and the column descriptions' name
and titles
atributes.
This allows for more flexibility, because columns in the CSV file may be
re-ordered without invalidating the metadata. A stricter matching can be forced
by specifying "header": false
and "skipRows": 1
in the table's dialect
description.However, csvw.CSVW
works correctly for
from the CSVW Test suites.
A CSVW-described dataset is basically equivalent to a Frictionless DataPackage where all
Data Resources are Tabular Data.
Thus, the csvw
package provides some conversion functionality. To
"read CSVW data from a Data Package", there's the csvw.TableGroup.from_frictionless_datapackage
method:
from csvw import TableGroup
tg = TableGroup.from_frictionless_datapackage('PATH/TO/datapackage.json')
To convert the metadata, the TableGroup
can then be serialzed:
tg.to_file('csvw-metadata.json')
Note that the CSVW metadata file must be written to the Data Package's directory to make sure relative paths to data resources work.
This functionality - together with the schema inference capabilities
of frictionless describe
- provides
a convenient way to bootstrap CSVW metadata for a set of "raw" CSV
files, implemented in the csvwdescribe
command described above.
This package is distributed under the Apache 2.0 license.