How to do Pipelines?

[mielvds]

The website says

Pipelines

Use SPARQL Results Sets as input for parametric queries. Decompose your data integration workflow

That's sounds interesting! But what do you mean exactly and how does it work? I currently use Apache Airflow to organise a decomposed mapping process, so anything in that direction is helpful.

[enridaga]

About the parametric queries, this is achieved by the CLI, see options -v, -i (the Results Sets as CSV), and -p (use parameters in output file names.

Maybe the best example is in this showcase project, for example:

fx -q imma-artist.sparql -i imma-artists.xml -p "artists/?artistNickname.jsonld" -f json

Where imma-artists.xml is a SPARQL result set.

[mielvds]

aha I see! Well too bad you can't do fx -q imma-artists.sparql -o imma-artists.xml -f xml | fx -q imma-artist.sparql - -p "artists/?artistNickname.jsonld" -f json :)

What I also find useful is splitting mapping queries in multiple queries, you know, for the conginitive load. Can you repeat -i to run multiple queries in sequence on the same data ?

[justin2004]

you could do something like the unix pipe by running insert queries instead of selects/constructs.

e.g.

curl --silent 'http://localhost:3000/sparql.anything'  \
--header "Accept: text/csv" \
--data-urlencode 'update=
PREFIX xyz: <http://sparql.xyz/facade-x/data/>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX fx: <http://sparql.xyz/facade-x/ns/>
PREFIX ex: <http://example.com/>
insert { graph ex:apple { ?s ?p ?o}}
WHERE {
 SERVICE <x-sparql-anything:> {
   fx:properties fx:location "https://www.google.com/search?q=friends" ;
                 fx:media-type "text/html" ;
                 fx:html.browser "firefox" .
   ?s ?p ?o .
 }
}'

then more inserts then eventually a select or construct...

curl --silent 'http://localhost:3000/sparql.anything'  \
--header "Accept: text/csv" \
--data-urlencode 'query=
PREFIX xyz: <http://sparql.xyz/facade-x/data/>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX fx: <http://sparql.xyz/facade-x/ns/>
PREFIX ex: <http://example.com/>
select *
WHERE {
   graph ex:apple {?s ?p ?o }
}'

you can think of the pipeline as just incrementally building named graphs.