| Tests naming convention |
| ----------------------- |
| Names of tests (suffixes of control.* files) are build from 2 or 3 words |
| separated by an underscore character. The first one is 'core' or 'ext', what |
| defines the set of PPD files to test. These two are described in the section |
| below. The second word of the name is either 'test' or 'dump'. The third and |
| optional word is 'debug'. All these three words influence the testing |
| procedure used by the autotest, their meanings are described in the next |
| paragraphs. |
| |
| Input sets of PPD files ('core' and 'ext') |
| ------------------------------------------ |
| There are two input sets of PPD files to test: 'core' and 'ext'. 'core' is the |
| smaller one and represents clusters defined in the file large_clusters.txt; it |
| is built by taking the first element (PPD filename) from every line of this |
| file. The set 'ext' is built in similar way by taking the first element (PPD |
| filename) from every line of small_clusters.txt file; however the elements |
| already present in 'core' are omitted. Currently, 'core' contains around ~130 |
| PPD files; the exact size equals the number of lines in large_clusters.txt. |
| The set 'ext' contains around ~1400 PPD files; the exact size equals the number |
| of lines in small_clusters.txt minus the number of lines in large_clusters.txt. |
| |
| Overall testing procedure |
| ------------------------- |
| The purpose of this autotest is to verify that given subset of PPD files work |
| in ChromeOS. Each PPD file is tested with the following procedure: |
| 1. A printer driver is added to CUPS server. |
| 2. Test documents are printed on the configured printer. |
| 3. Raw output from the CUPS server is intercepted by, so called, FakePrinter. |
| 4. CUPS logs are parsed to make sure that no errors occured. |
| 5. Obtained outputs are verified (see below) - test only. |
| 6. A printing pipeline used by CUPS is rerun in shell (see below) - debug only. |
| 7. All obtained outputs & logs are saved on the device (see below) - dump only. |
| 8. The printer driver is removed from CUPS server. |
| This procedure is repeated for every PPD file. The number of PPD files may be |
| large (~2K files). To decrease amount of time needed by the autotest, several |
| PPD files are tested simultaneously in parallel threads. Autotest always run |
| the procedure for all given PPD files and print a summary report at the end. |
| If at least one of PPD files fails, whole autotest is finished with failure |
| (but always all PPD files are processed). |
| |
| Output verification ('test') |
| ---------------------------- |
| Intercepted output is verified by comparision with the previous results |
| obtained for the same PPD. We cannot store outputs directly, because their |
| total size may have hundreds of megabytes. Instead of that short digest is |
| calculated for each obtained document and it is used for comparison. |
| A function for digests calculation is in the 'helpers.py' file. Not all |
| outputs can be tested this way because for some PPD files produced contents |
| differ between runs. List of PPD files for which we cannot calculate |
| constant digest is saved in the file digests_denylist.txt. Files with |
| expected digests for every test document are stored in the directory "digests". |
| If a digests for given pair (test document, PPD file) is missing, the test |
| checks only check if the output is not empty (or not too short). |
| |
| Rerun printing pipeline ('debug') |
| --------------------------------- |
| Every PPD file defines printing pipeline used by CUPS, consisting of external |
| binaries (so called filters). To make sure that every part of the printing |
| pipeline works correctly, it can be run outside CUPS in Linux shell. In tests |
| with 'debug' suffix in their names, whole pipeline is rerun outside CUPS. In |
| this mode every document is processed twice: the first time by CUPS and the |
| second time by extracted pipeline run as bash script. Additionally, each |
| pipeline's step is verified by checking its return code (it is supposed to be |
| equal 0) and the final output is compared with the output returned by CUPS |
| server. If the test is also defined as 'dump', each intermediate content is |
| saved to the directory given in path_outputs parameter. |
| This mode requires more disk space on tested device (~2GB in /tmp) and takes |
| more execution time (~2 times more). All information needed to reconstruct |
| a pipeline used by CUPS can are extracted from CUPS logs after the job is |
| completed. To do that, CUPS configuration must be modified: the LogLevel |
| option in /etc/cups/cupsd.conf must be set to "debug". This modification is |
| done when the autotest is initialized; the original setting is restored in |
| the cleaup stage. To make these modifications, 'debug' tests remount the root |
| partition in read-write mode. The partition is NOT remounted back to read-only |
| mode. |
| |
| Save outputs and logs ('dump') |
| ------------------------------ |
| All obtained outputs and logs are saved on the device in the directory |
| /tmp/PrinterPpds_outputs/. Results obtained from PPD files with the same prefix |
| are grouped together and stored in single archive to limit usage of disk space. |
| |
| Test parameters |
| --------------- |
| path_docs - path to directory with test documents (PDF files) |
| path_ppds - path to directory with PPD files, it is supposed to be compressed |
| as .tar.xz files (with a command "tar cJf ...") |
| path_digests - path to directory with files containing digests for |
| verification, if not set then outputs are not verified |
| path_outputs - if set, then all outputs are dumped there (given directory is |
| deleted if already exists); also all digests files are recalculated |
| and saved in the same directory |
| debug_mode - True or False, if True then for every test the printing pipeline |
| is extracted from CUPS logs and executed again outside CUPS |
| |
| Generating new digests |
| ---------------------- |
| The following procedure can be used to update digests: |
| 1. Run the test defined in control.all_dump: |
| test_that <device IP> platform_PrinterPpds.all_dump |
| 2. Download generated files with digests to your workstation |
| rsync root@<device IP>:/tmp/PrinterPpds_outputs/*.digests <local dir> |
| 3. Replace the files from the "digests" directory and commit changes |
| |
| Updating the archives with PPD files |
| ------------------------------------ |
| Currently, all tests are based on PPD files stored in local directories. The |
| autotest can download all PPD files by itself, but we do not use this option |
| to limit the number of possible points of failures during testing. Archives |
| with PPD files are prepared with ppdTool.go: |
| 1. Delete old files: |
| rm ppds_core.tar.xz ppds_ext.tar.xz large_clusters.txt small_clusters.txt |
| 2. Download all PPD files to ppds_all directory: |
| go run ppdTool.go download |
| 3. Calculate new clusters: |
| go run ppdTool.go compare |
| 4. Compress new directories with PPD files: |
| tar cJf ppds_core.tar.xz ppds_core |
| tar cJf ppds_ext.tar.xz ppds_ext |
| |
| Others |
| ------ |
| * How to get comma-separated list of all used ghostscript devices? |
| 1. Go to the directory generated by all_dump_debug |
| 2. Unpack all tar.xz archives (not on the device) |
| for d in *.pdf; |
| do |
| cd $d; |
| for f in *.tar.xz; do tar xf $f; done ; |
| cd ..; |
| done |
| 3. Run the following |
| grep -o sDEVICE=[^\ ]*\ ./*.pdf/*.err[12345] | cut -d \= -f 2 | sort | uniq | tr -d ' ' | tr '\n' ',' |
| 4. Add to the list uniprint (it is not caught by the command above) |
| * How to get a list of all used cups filters? |
| 1. Go to the directory generated by all_dump_debug |
| 2. Unpack all tar.xz archives (not on the device) |
| for d in *.pdf; |
| do |
| cd $d; |
| for f in *.tar.xz; do tar xf $f; done ; |
| cd ..; |
| done |
| 3. Run the following |
| grep -o '(exec -a "Fake[^"]*" [^ ]* ' ./*.pdf/*.sh | cut -d \ -f 4 | sort | uniq |