client/site_tests/platform_PrinterPpds/readme.txt - mirrors/cros/chromiumos/third_party/autotest - Git at Google

 Tests naming convention
 -----------------------
 Names of tests (suffixes of control.* files) are build from 2 or 3 words
 separated by an underscore character. The first one is '100' or 'all', what
 defines the set of PPD files to test: '100' means 100 chosen (most often used)
 PPD files (stored in the archive ppds_100) while 'all' means all available PPD
 files (stored in the archive ppds_all). The second word of the name is either
 'test' or 'dump'. The third and optional word is 'debug'. All these three words
 influence testing procedure used by the autotest, their meanings are described
 in next paragraphs.

 Overall testing procedure
 -------------------------
 The purpose of this autotest is to verify that given subset of PPD files work
 in ChromeOS. Each PPD file is tested with the following procedure:
 1. A printer driver is added to CUPS server.
 2. Test documents are printed on the configured printer.
 3. Raw output from the CUPS server is intercepted by, so called, FakePrinter.
 4. CUPS logs are parsed to make sure that no errors occured.
 5. Obtained outputs are verified (see below) - test only.
 6. A printing pipeline used by CUPS is rerun in shell (see below) - debug only.
 7. All obtained outputs & logs are saved on the device (see below) - dump only.
 8. The printer driver is removed from CUPS server.
 This procedure is repeated for every PPD file. The number of PPD files may be
 large (3K+ files). To decrease amount of time needed by the autotest, several
 PPD files are tested simultaneously in parallel threads. Autotest always run
 the procedure for all given PPD files and print a summary report at the end.
 If at least one of PPD files fails, whole autotest is finished with failure
 (but always all PPD files are processed).

 Output verification ('test')
 ----------------------------
 Intercepted output is verified by comparision with the previous results
 obtained for the same PPD. We cannot store outputs directly, because their
 total size may have hundreds of megabytes. Instead of that short digest is
 calculated for each obtained document and it is used for comparison.
 A function for digests calculation is in the 'helpers.py' file. Not all
 outputs can be tested this way because for some PPD files produced contents
 differ between runs. List of PPD files for which we cannot calculate
 constant digest is saved in the file digests_blacklist.txt. Files with
 expected digests for every test document are stored in the directory "digests".
 If a digests for given pair (test document, PPD file) is missing, the test
 checks only check if the output is not empty (or not too short).

 Rerun printing pipeline ('debug')
 ---------------------------------
 Every PPD file defines printing pipeline used by CUPS, consisting of external
 binaries (so called filters). To make sure that every part of the printing
 pipeline works correctly, it can be run outside CUPS in Linux shell. In tests
 with 'debug' suffix in their names, whole pipeline is rerun outside CUPS. In
 this mode every document is processed twice: the first time by CUPS and the
 second time by extracted pipeline run as bash script. Additionally, each
 pipeline's step is verified by checking its return code (it is supposed to be
 equal 0) and the final output is compared with the output returned by CUPS
 server. If the test is also defined as 'dump', each intermediate content is
 saved to the directory given in path_outputs parameter.
 This mode requires more disk space on tested device (~2GB in /tmp) and takes
 more execution time (~2 times more). All information needed to reconstruct
 a pipeline used by CUPS can are extracted from CUPS logs after the job is
 completed. To do that, CUPS configuration must be modified: the LogLevel
 option in /etc/cups/cupsd.conf must be set to "debug". This modification is
 done when the autotest is initialized; the original setting is restored in
 the cleaup stage.


 Save outputs and logs ('dump')
 ------------------------------
 All obtained outputs and logs are saved on the device in the directory
 /tmp/PrinterPpds_outputs/. Results obtained from PPD files with the same prefix
 are grouped together and stored in single archive to limit usage of disk space.

 Test parameters
 ---------------
 path_docs - path to directory with test documents (PDF files)
 path_ppds - path to directory with PPD files, if not set then all available PPD
             files are downloaded and tested
 path_digests - path to directory with files containing digests for
             verification, if not set then outputs are not verified
 path_outputs - if set, then all outputs are dumped there (given directory is
             deleted if already exists); also all digests files are recalculated
             and saved in the same directory
 debug_mode - True or False, if True then for every test the printing pipeline
             is extracted from CUPS logs and executed again outside CUPS

 Generating new digests
 ----------------------
 The following procedure can be used to update digests:
 1. Run the test defined in control.all_dump:
         test_that <device IP>  platform_PrinterPpds.all_dump
 2. Download generated files with digests to your workstation
         rsync root@<device IP>:/tmp/PrinterPpds_outputs/*.digests <local dir>
 3. Replace the files from the "digests" directory and commit changes

 Updating the archive with PPD files
 -----------------------------------
 Currently, all tests are based on PPD files stored in local directories. The
 autotest can download all PPD files by itself, but we do not use this option
 to limit the number of possible points of failures during testing. PPD files
 are stored in the archive 'ppds_all.tar.xz'. To replace the archive with the
 current list of supported PPD files, one can use the script
 'download_ppds_make_archive.py'.

 Others
 ------
 * How to get comma-separated list of all used ghostscript devices?
   1. Go to the directory generated by all_dump_debug
   2. Unpack all tar.xz archives (not on the device)
         for d in *.pdf;
         do
             cd $d;
             for f in *.tar.xz; do tar xf $f; done ;
             cd ..;
         done
   3. Run the following
         grep -o sDEVICE=[^\ ]*\  ./*.pdf/*.err[12345] | cut -d \= -f 2 | sort | uniq | tr -d ' ' | tr '\n' ','
   4. Add to the list uniprint (it is not caught by the command above)
 * How to get a list of all used cups filters?
   1. Go to the directory generated by all_dump_debug
   2. Unpack all tar.xz archives (not on the device)
         for d in *.pdf;
         do
             cd $d;
             for f in *.tar.xz; do tar xf $f; done ;
             cd ..;
         done
   3. Run the following
         grep -o '(exec -a "Fake[^"]*" [^ ]* ' ./*.pdf/*.sh | cut -d \  -f 4 | sort | uniq
	Tests naming convention
	-----------------------
	Names of tests (suffixes of control.* files) are build from 2 or 3 words
	separated by an underscore character. The first one is '100' or 'all', what
	defines the set of PPD files to test: '100' means 100 chosen (most often used)
	PPD files (stored in the archive ppds_100) while 'all' means all available PPD
	files (stored in the archive ppds_all). The second word of the name is either
	'test' or 'dump'. The third and optional word is 'debug'. All these three words
	influence testing procedure used by the autotest, their meanings are described
	in next paragraphs.

	Overall testing procedure
	-------------------------
	The purpose of this autotest is to verify that given subset of PPD files work
	in ChromeOS. Each PPD file is tested with the following procedure:
	1. A printer driver is added to CUPS server.
	2. Test documents are printed on the configured printer.
	3. Raw output from the CUPS server is intercepted by, so called, FakePrinter.
	4. CUPS logs are parsed to make sure that no errors occured.
	5. Obtained outputs are verified (see below) - test only.
	6. A printing pipeline used by CUPS is rerun in shell (see below) - debug only.
	7. All obtained outputs & logs are saved on the device (see below) - dump only.
	8. The printer driver is removed from CUPS server.
	This procedure is repeated for every PPD file. The number of PPD files may be
	large (3K+ files). To decrease amount of time needed by the autotest, several
	PPD files are tested simultaneously in parallel threads. Autotest always run
	the procedure for all given PPD files and print a summary report at the end.
	If at least one of PPD files fails, whole autotest is finished with failure
	(but always all PPD files are processed).

	Output verification ('test')
	----------------------------
	Intercepted output is verified by comparision with the previous results
	obtained for the same PPD. We cannot store outputs directly, because their
	total size may have hundreds of megabytes. Instead of that short digest is
	calculated for each obtained document and it is used for comparison.
	A function for digests calculation is in the 'helpers.py' file. Not all
	outputs can be tested this way because for some PPD files produced contents
	differ between runs. List of PPD files for which we cannot calculate
	constant digest is saved in the file digests_blacklist.txt. Files with
	expected digests for every test document are stored in the directory "digests".
	If a digests for given pair (test document, PPD file) is missing, the test
	checks only check if the output is not empty (or not too short).

	Rerun printing pipeline ('debug')
	---------------------------------
	Every PPD file defines printing pipeline used by CUPS, consisting of external
	binaries (so called filters). To make sure that every part of the printing
	pipeline works correctly, it can be run outside CUPS in Linux shell. In tests
	with 'debug' suffix in their names, whole pipeline is rerun outside CUPS. In
	this mode every document is processed twice: the first time by CUPS and the
	second time by extracted pipeline run as bash script. Additionally, each
	pipeline's step is verified by checking its return code (it is supposed to be
	equal 0) and the final output is compared with the output returned by CUPS
	server. If the test is also defined as 'dump', each intermediate content is
	saved to the directory given in path_outputs parameter.
	This mode requires more disk space on tested device (~2GB in /tmp) and takes
	more execution time (~2 times more). All information needed to reconstruct
	a pipeline used by CUPS can are extracted from CUPS logs after the job is
	completed. To do that, CUPS configuration must be modified: the LogLevel
	option in /etc/cups/cupsd.conf must be set to "debug". This modification is
	done when the autotest is initialized; the original setting is restored in
	the cleaup stage.


	Save outputs and logs ('dump')
	------------------------------
	All obtained outputs and logs are saved on the device in the directory
	/tmp/PrinterPpds_outputs/. Results obtained from PPD files with the same prefix
	are grouped together and stored in single archive to limit usage of disk space.

	Test parameters
	---------------
	path_docs - path to directory with test documents (PDF files)
	path_ppds - path to directory with PPD files, if not set then all available PPD
	files are downloaded and tested
	path_digests - path to directory with files containing digests for
	verification, if not set then outputs are not verified
	path_outputs - if set, then all outputs are dumped there (given directory is
	deleted if already exists); also all digests files are recalculated
	and saved in the same directory
	debug_mode - True or False, if True then for every test the printing pipeline
	is extracted from CUPS logs and executed again outside CUPS

	Generating new digests
	----------------------
	The following procedure can be used to update digests:
	1. Run the test defined in control.all_dump:
	test_that <device IP> platform_PrinterPpds.all_dump
	2. Download generated files with digests to your workstation
	rsync root@<device IP>:/tmp/PrinterPpds_outputs/*.digests <local dir>
	3. Replace the files from the "digests" directory and commit changes

	Updating the archive with PPD files
	-----------------------------------
	Currently, all tests are based on PPD files stored in local directories. The
	autotest can download all PPD files by itself, but we do not use this option
	to limit the number of possible points of failures during testing. PPD files
	are stored in the archive 'ppds_all.tar.xz'. To replace the archive with the
	current list of supported PPD files, one can use the script
	'download_ppds_make_archive.py'.

	Others
	------
	* How to get comma-separated list of all used ghostscript devices?
	1. Go to the directory generated by all_dump_debug
	2. Unpack all tar.xz archives (not on the device)
	for d in *.pdf;
	do
	cd $d;
	for f in *.tar.xz; do tar xf $f; done ;
	cd ..;
	done
	3. Run the following
	grep -o sDEVICE=[^\ ]\ ./.pdf/*.err[12345] \| cut -d \= -f 2 \| sort \| uniq \| tr -d ' ' \| tr '\n' ','
	4. Add to the list uniprint (it is not caught by the command above)
	* How to get a list of all used cups filters?
	1. Go to the directory generated by all_dump_debug
	2. Unpack all tar.xz archives (not on the device)
	for d in *.pdf;
	do
	cd $d;
	for f in *.tar.xz; do tar xf $f; done ;
	cd ..;
	done
	3. Run the following
	grep -o '(exec -a "Fake[^"]" [^ ] ' ./.pdf/.sh \| cut -d \ -f 4 \| sort \| uniq