Data sources

Prepare your data directory

I suggest to create a data directory under the course working directory, like shown HERE. If you have not retrieved to whole data tree, create a directory, e.g. ~/mpe2018/Data, and copy there the files you find in the web directory /imprs-db/Data.

Note:
To download on your PC all the data files in the Data directory use the following wget
command:

shell> cd ~/mpe2018
shell> wget -r -l1 -np -nH -nc --cut-dirs=1 -A html,gz,sql,csv,tsv,MYD,MYI,frm https://astrordbms.oas.inaf.it/imprs-db/Data

shell> ls
shell> ls Data
shell> rm index.html

Note that if you want existing local file to be overwritten, then you have to remove the -nc option! Otherwise just remove those you want to retrieve again. Can also just replace -nc with -N to download only files newer on the server with respect to your local ones.

Note:
To download only the data files with a given extension from the above Data directory do as follows (this is to download all the .sql files:

shell> cd ~/mpe2018
shell> wget -r -l1 -np -nH --cut-dirs=1 -A.sql https://astrordbms.oas.inaf.it/imprs-db/Data

shell> ls Data
bsc.sql                        load_3XMM_DR4cat_v1.0.sql      simpleBSC.sql
bsc_1r.sql                     load_3xmmdr8cat.sql            simpleBSC_1r.sql
csc_query.sql                  load_3xmmdr8cat_slim.sql       world.sql
load_3XMM_DR4cat_slim_v1.1.sql messier.sql                    world_innodb.sql

Similarly for the files with another extension, just replace the sql after A. with the extension you want, e.g. gz to download all the gzipped files.

Note:
For convenience, the tar file soft_dirs.tar.gz contains the whole Soft tree structure.
If not done yet, download it in the ~/mpe2018 directory (or another you prefer) and untar it:

shell> cd ~/mpe2018
shell> wget https://astrordbms.oas.inaf.it/imprs-db/soft_dirs.tar.gz
shell> tar zxvf soft_dirs.tar.gz

Local data sources

These files are in various formats: csv, tsv, vot, fits, sql, MyISAM. Note that some .sql files are simple table creation commands, whereas others are MySQL dump files ready to be executed to create and import tables. Other SQL scripts are in /imprs-db/Soft/SQL/.

Data formats:

CSV – comma separated values (text): the field separator is a ,

TSV – tab separated values (text): the field separator is a Tab (also referred as ^I or \t)

VOT – VOTable format (text): a Virtual Observatory specific XML file

FITS – the standard format for astronomy (header text, data binary): only 1-d tables allowed.

SQL – Structured Query Language (text): comment lines start with “-- ” or "# "

MyISAM – a triplet of MySQL table data (binary): .MYD, MYI, .frm

Some of these files, in particular the VOTable files, were downloaded by the sites listed in next section.

External data resources

VOTable reference sites:
https://www.ivoa.net/documents/VOTable/20130315/PR-VOTable-1.3-20130315.html#ToC7

https://www.ivoa.net/documents/VOTable/

Some example queries on the Chandra archive (split on multiple lines for clarity – see web site for more info):

shell>
  curl -o cx_out.vot --form coordFormat=decimal --form outputFormat=vot \
       --form query='SELECT TOP 50 m.name, m.ra, m.dec, m.significance, m.flux_aper_b, m.alpha \
       FROM master_source m WHERE (m.significance > 10.0 AND m.pileup_flag = 0)' \
      'https://cda.cfa.harvard.ed u/csccli/getProperties'

  curl -o cx_out.txt --form coordFormat=decimal --form query=@csc_query.sql \
       'https://cda.cfa.harvard.edu/csccli/getProperties'

In the first query we ask for an output file in VOTable format, while in the second we’ll get a (default) text file with fields separated by the tab character. As these are also referred as tsv files (Tab Separated Values), we could have used cx_out.tsv as output file name. The tab is the default field delimiter when importing text files in MySQL!
Also note how in the second command we use a local query contained in the file csc_query.sql. More example queries:

SELECT DISTINCT m.name, m.ra, m.dec, m.extent_flag, m.err_ellipse_r0, m.err_ellipse_r1,
   m.conf_flag, m.flux_aper_b, m.flux_aper_w, m.significance
   FROM master_source m WHERE (m.significance > 10.0 AND m.pileup_flag = 0)

SELECT DISTINCT top 1000 m.name, m.ra, m.dec, m.err_ellipse_r0, m.conf_flag, m.sat_src_flag,
  m.significance,m.flux_aper_b,m.flux_aper_w
  FROM master_source m WHERE (m.significance > 10.0 AND m.pileup_flag = 0)
  ORDER BY name ASC