;;; This file is a trace of running the universal tester to compare different ;;; learning systems by automatically generating learning curves. ;;; Subsequently added comments. > (cd "/u/mooney/ml-progs") ; switch to ml-progs directory #P"/tmp_mnt/v/sally/v1/mooney/ml-progs/" > (load "universal-tester") ; load the universal tester ;;; Loading binary file "universal-tester.sbin" ;;; Loading binary file "/u/mooney/ml-progs/data-utilities.sbin" ;;; Loading binary file "/u/mooney/ml-progs/t-test.sbin" #P"/tmp_mnt/v/sally/v1/mooney/ml-progs/universal-tester.sbin" > (load "id3-all") ; load the ID3 system for inducing decision trees ;;; Loading binary file "id3-all.sbin" #P"/tmp_mnt/v/sally/v1/mooney/ml-progs/id3-all.sbin" > (load "pfoil") ; load the propositional FOIL system for inducing rules ;;; Loading binary file "pfoil.sbin" #P"/tmp_mnt/v/sally/v1/mooney/ml-progs/pfoil.sbin" ;;; First make a file for storing the exact training/test splits to be run so ;;; that later we can run additional experiments on exactly the same training ;;; and test examples. "sample-dna-saved-tests.lisp" is the name of the file ;;; we want to save the test information in. 10 is the number of random trials ;;; we want to run. 80 is the total number of training examples we want to use. ;;; (10 25 40 80) is the set of points (in number of training examples) we want ;;; have plotted on the learning curve (alternatively you can specify an ;;; increment like 10 for plotting after every 10 examples). NIL means to ;;; default the number of test examples to using all the remaining examples in ;;; the data file (otherwise give a number). "/u/mooney/data/dna-standard.lisp" ;;; is the file to use as the data (106 examples for DNA Promoter recognition). > (make-saved-tests "sample-dna-saved-tests.lisp" 10 80 '(10 25 40 80) nil "/u/mooney/data/dna-standard.lisp") ;;; Loading source file "/tmp_mnt/v/sally/v1/mooney/data/dna-standard.lisp" ;;; Warning: File "/tmp_mnt/v/sally/v1/mooney/data/dna-standard.lisp" does not begin with IN-PACKAGE. Loading into package "USER" NIL ;;; Now we run tests to compare ID3 and PFOIL using these saved tests. ;;; (id3 pfoil) is the list of systems we want to compare. ;;; "sample-dna-saved-tests.lisp" is the saved test data file. ;;; "sample-dna-results" is the file in which to store the test results. > (run-saved-tests '(id3 pfoil) "sample-dna-saved-tests.lisp" "sample-dna-results") ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 1: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 34.62% Training Time: 0.01 sec Test Time: 0.00 sec Concept Complexity: 1 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 34.62% Training Time: 0.00 sec Test Time: 0.00 sec Concept Complexity: 0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 53.85% Training Time: 0.28 sec Test Time: 0.00 sec Concept Complexity: 4 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 80.77% Training Time: 0.04 sec Test Time: 0.00 sec Concept Complexity: 1 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 53.85% Training Time: 0.37 sec Test Time: 0.00 sec Concept Complexity: 7 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 84.62% Training Time: 0.12 sec Test Time: 0.01 sec Concept Complexity: 3 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 69.23% Training Time: 0.93 sec Test Time: 0.01 sec Concept Complexity: 16 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 73.08% Training Time: 0.26 sec Test Time: 0.01 sec Concept Complexity: 6 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 73.08% Training Time: 1.50 sec Test Time: 0.00 sec Concept Complexity: 25 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 69.23% Training Time: 0.62 sec Test Time: 0.00 sec Concept Complexity: 11 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 2: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 46.15% Training Time: 0.00 sec Test Time: 0.00 sec Concept Complexity: 1 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 53.85% Training Time: 0.00 sec Test Time: 0.00 sec Concept Complexity: 0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 46.15% Training Time: 0.30 sec Test Time: 0.00 sec Concept Complexity: 7 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 69.23% Training Time: 0.05 sec Test Time: 0.00 sec Concept Complexity: 2 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 0.54 sec Test Time: 0.00 sec Concept Complexity: 10 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 73.08% Training Time: 0.15 sec Test Time: 0.01 sec Concept Complexity: 4 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 61.54% Training Time: 0.67 sec Test Time: 0.01 sec Concept Complexity: 10 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 0.26 sec Test Time: 0.00 sec Concept Complexity: 6 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 80.77% Training Time: 1.41 sec Test Time: 0.00 sec Concept Complexity: 22 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 73.08% Training Time: 0.67 sec Test Time: 0.01 sec Concept Complexity: 12 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 3: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 61.54% Training Time: 0.00 sec Test Time: 0.00 sec Concept Complexity: 1 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 38.46% Training Time: 0.00 sec Test Time: 0.00 sec Concept Complexity: 0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 42.31% Training Time: 0.28 sec Test Time: 0.01 sec Concept Complexity: 7 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 46.15% Training Time: 0.05 sec Test Time: 0.00 sec Concept Complexity: 2 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 0.38 sec Test Time: 0.00 sec Concept Complexity: 7 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 61.54% Training Time: 0.11 sec Test Time: 0.01 sec Concept Complexity: 3 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 69.23% Training Time: 0.91 sec Test Time: 0.01 sec Concept Complexity: 16 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 65.38% Training Time: 0.34 sec Test Time: 0.01 sec Concept Complexity: 9 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 88.46% Training Time: 1.73 sec Test Time: 0.00 sec Concept Complexity: 28 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 80.77% Training Time: 0.64 sec Test Time: 0.01 sec Concept Complexity: 12 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 4: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 57.69% Training Time: 0.00 sec Test Time: 0.00 sec Concept Complexity: 1 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 57.69% Training Time: 0.00 sec Test Time: 0.00 sec Concept Complexity: 0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 0.19 sec Test Time: 0.00 sec Concept Complexity: 4 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 0.04 sec Test Time: 0.00 sec Concept Complexity: 1 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 65.38% Training Time: 0.56 sec Test Time: 0.00 sec Concept Complexity: 10 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 73.08% Training Time: 0.18 sec Test Time: 0.00 sec Concept Complexity: 5 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 65.38% Training Time: 0.64 sec Test Time: 0.01 sec Concept Complexity: 10 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 73.08% Training Time: 0.30 sec Test Time: 0.00 sec Concept Complexity: 8 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 1.60 sec Test Time: 0.00 sec Concept Complexity: 25 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 0.74 sec Test Time: 0.01 sec Concept Complexity: 14 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 5: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 30.77% Training Time: 0.00 sec Test Time: 0.01 sec Concept Complexity: 1 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 69.23% Training Time: 0.00 sec Test Time: 0.00 sec Concept Complexity: 0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 80.77% Training Time: 0.30 sec Test Time: 0.00 sec Concept Complexity: 7 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 73.08% Training Time: 0.05 sec Test Time: 0.01 sec Concept Complexity: 2 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 84.62% Training Time: 0.52 sec Test Time: 0.00 sec Concept Complexity: 10 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 73.08% Training Time: 0.16 sec Test Time: 0.00 sec Concept Complexity: 4 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 84.62% Training Time: 0.76 sec Test Time: 0.01 sec Concept Complexity: 13 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 73.08% Training Time: 0.32 sec Test Time: 0.00 sec Concept Complexity: 7 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 80.77% Training Time: 1.67 sec Test Time: 0.00 sec Concept Complexity: 28 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 84.62% Training Time: 0.71 sec Test Time: 0.01 sec Concept Complexity: 13 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 6: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 57.69% Training Time: 0.00 sec Test Time: 0.00 sec Concept Complexity: 1 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 57.69% Training Time: 0.00 sec Test Time: 0.00 sec Concept Complexity: 0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 50.00% Training Time: 0.28 sec Test Time: 0.00 sec Concept Complexity: 7 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 57.69% Training Time: 0.05 sec Test Time: 0.00 sec Concept Complexity: 2 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 65.38% Training Time: 0.55 sec Test Time: 0.00 sec Concept Complexity: 10 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 92.31% Training Time: 0.15 sec Test Time: 0.00 sec Concept Complexity: 4 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 65.38% Training Time: 0.83 sec Test Time: 0.01 sec Concept Complexity: 16 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 0.23 sec Test Time: 0.01 sec Concept Complexity: 6 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 73.08% Training Time: 1.49 sec Test Time: 0.00 sec Concept Complexity: 22 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 0.71 sec Test Time: 0.01 sec Concept Complexity: 13 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 7: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 46.15% Training Time: 0.00 sec Test Time: 0.01 sec Concept Complexity: 1 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 53.85% Training Time: 0.00 sec Test Time: 0.01 sec Concept Complexity: 0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 46.15% Training Time: 0.17 sec Test Time: 0.00 sec Concept Complexity: 4 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 0.05 sec Test Time: 0.00 sec Concept Complexity: 2 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 73.08% Training Time: 0.64 sec Test Time: 0.00 sec Concept Complexity: 13 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 80.77% Training Time: 0.16 sec Test Time: 0.01 sec Concept Complexity: 4 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 69.23% Training Time: 0.95 sec Test Time: 0.01 sec Concept Complexity: 16 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 0.31 sec Test Time: 0.01 sec Concept Complexity: 8 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 1.84 sec Test Time: 0.00 sec Concept Complexity: 31 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 80.77% Training Time: 0.68 sec Test Time: 0.00 sec Concept Complexity: 11 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 8: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 42.31% Training Time: 0.00 sec Test Time: 0.00 sec Concept Complexity: 1 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 42.31% Training Time: 0.00 sec Test Time: 0.00 sec Concept Complexity: 0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 46.15% Training Time: 0.18 sec Test Time: 0.01 sec Concept Complexity: 4 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 73.08% Training Time: 0.03 sec Test Time: 0.00 sec Concept Complexity: 1 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 57.69% Training Time: 0.54 sec Test Time: 0.00 sec Concept Complexity: 10 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 57.69% Training Time: 0.15 sec Test Time: 0.01 sec Concept Complexity: 4 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 80.77% Training Time: 0.86 sec Test Time: 0.00 sec Concept Complexity: 16 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 73.08% Training Time: 0.22 sec Test Time: 0.00 sec Concept Complexity: 5 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 80.77% Training Time: 1.57 sec Test Time: 0.00 sec Concept Complexity: 25 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 84.62% Training Time: 0.53 sec Test Time: 0.01 sec Concept Complexity: 10 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 9: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 53.85% Training Time: 0.00 sec Test Time: 0.01 sec Concept Complexity: 1 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 53.85% Training Time: 0.00 sec Test Time: 0.01 sec Concept Complexity: 0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 69.23% Training Time: 0.18 sec Test Time: 0.01 sec Concept Complexity: 4 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 73.08% Training Time: 0.04 sec Test Time: 0.00 sec Concept Complexity: 2 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 0.53 sec Test Time: 0.00 sec Concept Complexity: 10 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 80.77% Training Time: 0.15 sec Test Time: 0.00 sec Concept Complexity: 4 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 0.77 sec Test Time: 0.01 sec Concept Complexity: 13 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 73.08% Training Time: 0.31 sec Test Time: 0.01 sec Concept Complexity: 8 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 1.61 sec Test Time: 0.01 sec Concept Complexity: 25 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 73.08% Training Time: 0.72 sec Test Time: 0.01 sec Concept Complexity: 13 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 10: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 50.00% Training Time: 0.00 sec Test Time: 0.00 sec Concept Complexity: 1 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 50.00% Training Time: 0.00 sec Test Time: 0.00 sec Concept Complexity: 0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 53.85% Training Time: 0.19 sec Test Time: 0.01 sec Concept Complexity: 4 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 53.85% Training Time: 0.04 sec Test Time: 0.00 sec Concept Complexity: 1 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 84.62% Training Time: 0.53 sec Test Time: 0.00 sec Concept Complexity: 10 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 69.23% Training Time: 0.16 sec Test Time: 0.01 sec Concept Complexity: 4 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 69.23% Training Time: 0.87 sec Test Time: 0.00 sec Concept Complexity: 16 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 84.62% Training Time: 0.33 sec Test Time: 0.01 sec Concept Complexity: 8 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 53.85% Training Time: 1.89 sec Test Time: 0.00 sec Concept Complexity: 31 Training PFOIL...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 73.08% Training Time: 0.66 sec Test Time: 0.00 sec Concept Complexity: 11 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ NIL ;;; Now we want to run a statistical t-test to compare the overall results. ;;; Compares test-accuracy by default and uses 2-tailed test to be conservative. > (t-test-file "sample-dna-results") Total number of samples in data: 10 Comparing ID3 to PFOIL: For 0 training examples: Analysis of ID3 mean = 48.077 (sd 9.615) versus PFOIL mean = 51.154 (sd 9.738). Difference: -3.077 Paired t-test results: t = -0.647, df = 9 (not significant for 2-tailed test) The 0.05 level confidence interval on the difference between the means: -13.834 to 7.680 For 10 training examples: Analysis of ID3 mean = 56.538 (sd 13.218) versus PFOIL mean = 68.077 (sd 10.885). Difference: -11.538* Paired t-test results: t = -2.620, df = 9 (significant at the 0.05 level for 2-tailed test) The 0.05 level confidence interval on the difference between the means: -21.500 to -1.577 For 25 training examples: Analysis of ID3 mean = 71.538 (sd 10.059) versus PFOIL mean = 74.615 (sd 9.911). Difference: -3.077 Paired t-test results: t = -0.605, df = 9 (not significant for 2-tailed test) The 0.05 level confidence interval on the difference between the means: -14.590 to 8.436 For 40 training examples: Analysis of ID3 mean = 71.154 (sd 6.934) versus PFOIL mean = 74.615 (sd 4.615). Difference: -3.462 Paired t-test results: t = -1.132, df = 9 (not significant for 2-tailed test) The 0.05 level confidence interval on the difference between the means: -10.379 to 3.456 For 80 training examples: Analysis of ID3 mean = 76.154 (sd 8.566) versus PFOIL mean = 77.308 (sd 5.000). Difference: -1.154 Paired t-test results: t = -0.461, df = 9 (not significant for 2-tailed test) The 0.05 level confidence interval on the difference between the means: -6.814 to 4.507 NIL ;;; Make a plot file suitable for the XGRAPH system for generating graphs. ;;; This creates a file sample-dna-results.test-accuracy-plot in the ;;; current directory that can be displayed by calling: ;;; xgraph sample-dna-results.test-accuracy-plot in UNIX > (make-plot-file "sample-dna-results") NIL ;;; We can also make a graph for training time. ;;; This creates a file sample-dna-results.train-time-plot ;;; (train-accuracy test-accuracy train-time test-time concept-complexity) ;;; are the different dependent variables we can plot. > (make-plot-file "sample-dna-results" 'train-time) NIL ;;; We can also use a 1-tailed test to compare training-time. ;;; (train-accuracy test-accuracy train-time test-time concept-complexity) ;;; are the list of dependent variables we can compare. > (t-test-file "sample-dna-results" 1 'train-time) Total number of samples in data: 10 Comparing ID3 to PFOIL: For 0 training examples: Analysis of ID3 mean = 0.001 (sd 0.003) versus PFOIL mean = 0.000 (sd 0.000). Difference: 0.001 Paired t-test results: t = 1.000, df = 9 (not significant for 1-tailed test) The 0.05 level confidence interval on the difference between the means: -0.001 to 0.003 For 10 training examples: Analysis of ID3 mean = 0.235 (sd 0.054) versus PFOIL mean = 0.044 (sd 0.007). Difference: 0.191* Paired t-test results: t = 11.439, df = 9 (significant at the 0.0005 level for 1-tailed test) The 0.05 level confidence interval on the difference between the means: 0.160 to 0.222 For 25 training examples: Analysis of ID3 mean = 0.516 (sd 0.077) versus PFOIL mean = 0.149 (sd 0.019). Difference: 0.367* Paired t-test results: t = 17.765, df = 9 (significant at the 0.0005 level for 1-tailed test) The 0.05 level confidence interval on the difference between the means: 0.329 to 0.405 For 40 training examples: Analysis of ID3 mean = 0.819 (sd 0.101) versus PFOIL mean = 0.288 (sd 0.040). Difference: 0.531* Paired t-test results: t = 14.927, df = 9 (significant at the 0.0005 level for 1-tailed test) The 0.05 level confidence interval on the difference between the means: 0.466 to 0.596 For 80 training examples: Analysis of ID3 mean = 1.631 (sd 0.146) versus PFOIL mean = 0.668 (sd 0.058). Difference: 0.963* Paired t-test results: t = 18.718, df = 9 (significant at the 0.0005 level for 1-tailed test) The 0.05 level confidence interval on the difference between the means: 0.869 to 1.057 NIL ;;; Later we decide we would like to run the same tests on the Nearest Neighbor ;;; instance-based method. > (load "knn") ; Load K Nearest Neighbor method (K=1 by default) ;;; Loading binary file "knn.sbin" #P"/tmp_mnt/v/sally/v1/mooney/ml-progs/knn.sbin" ;;; Run the same tests on KNN and save in the file "sample-dna-knn-results" > (run-saved-tests '(knn) "sample-dna-saved-tests.lisp" "sample-dna-knn-results") ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 1: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 38.46% Training Time: 0.00 sec Test Time: 0.00 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 0.00 sec Test Time: 0.28 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 0.00 sec Test Time: 0.71 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 0.00 sec Test Time: 1.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 0.00 sec Test Time: 2.15 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 2: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 38.46% Training Time: 0.00 sec Test Time: 0.00 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 73.08% Training Time: 0.00 sec Test Time: 0.26 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 80.77% Training Time: 0.00 sec Test Time: 0.67 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 80.77% Training Time: 0.00 sec Test Time: 1.07 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 84.62% Training Time: 0.00 sec Test Time: 2.12 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 3: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 50.00% Training Time: 0.00 sec Test Time: 0.01 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 84.62% Training Time: 0.00 sec Test Time: 0.27 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 92.31% Training Time: 0.00 sec Test Time: 0.67 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 96.15% Training Time: 0.00 sec Test Time: 1.06 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 96.15% Training Time: 0.00 sec Test Time: 2.13 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 4: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 57.69% Training Time: 0.00 sec Test Time: 0.00 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 65.38% Training Time: 0.00 sec Test Time: 0.26 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 84.62% Training Time: 0.00 sec Test Time: 0.67 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 84.62% Training Time: 0.00 sec Test Time: 1.08 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 80.77% Training Time: 0.00 sec Test Time: 2.15 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 5: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 46.15% Training Time: 0.00 sec Test Time: 0.00 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 50.00% Training Time: 0.00 sec Test Time: 0.26 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 53.85% Training Time: 0.00 sec Test Time: 0.67 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 61.54% Training Time: 0.00 sec Test Time: 1.06 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 65.38% Training Time: 0.00 sec Test Time: 2.14 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 6: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 34.62% Training Time: 0.00 sec Test Time: 0.00 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 80.77% Training Time: 0.00 sec Test Time: 0.27 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 88.46% Training Time: 0.00 sec Test Time: 0.68 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 88.46% Training Time: 0.00 sec Test Time: 1.07 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 80.77% Training Time: 0.00 sec Test Time: 2.16 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 7: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 61.54% Training Time: 0.00 sec Test Time: 0.00 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 53.85% Training Time: 0.00 sec Test Time: 0.27 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 61.54% Training Time: 0.00 sec Test Time: 0.67 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 57.69% Training Time: 0.00 sec Test Time: 1.12 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 73.08% Training Time: 0.00 sec Test Time: 2.13 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 8: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 61.54% Training Time: 0.00 sec Test Time: 0.00 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 80.77% Training Time: 0.00 sec Test Time: 0.27 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 84.62% Training Time: 0.00 sec Test Time: 0.68 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 80.77% Training Time: 0.00 sec Test Time: 1.09 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 73.08% Training Time: 0.00 sec Test Time: 2.14 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 9: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 61.54% Training Time: 0.00 sec Test Time: 0.01 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 69.23% Training Time: 0.00 sec Test Time: 0.27 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 65.38% Training Time: 0.00 sec Test Time: 0.67 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 69.23% Training Time: 0.00 sec Test Time: 1.08 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 80.77% Training Time: 0.00 sec Test Time: 2.12 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 10: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 46.15% Training Time: 0.00 sec Test Time: 0.01 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 61.54% Training Time: 0.00 sec Test Time: 0.27 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 69.23% Training Time: 0.00 sec Test Time: 0.67 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 73.08% Training Time: 0.00 sec Test Time: 1.08 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training KNN...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 84.62% Training Time: 0.00 sec Test Time: 2.13 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ NIL ;;; In order to compare the new results to the old, we first change the old result ;;; file to add the new results to it. If you don't want to overwrite the old result ;;; file use (combine-test-results file1 file2 new-file) to combine the two result ;;; files into a new file. > (append-test-results "sample-dna-results" "sample-dna-knn-results") #P"/tmp_mnt/v/sally/v1/mooney/ml-progs/sample-dna-results" #P"/tmp_mnt/v/sally/v1/mooney/ml-progs/temp" #P"/tmp_mnt/v/sally/v1/mooney/ml-progs/sample-dna-results" ;;; Now rerun the t-test, which will compare every pair of systems in the ;;; new version of the result file > (t-test-file "sample-dna-results" 1) Total number of samples in data: 10 Comparing ID3 to PFOIL: For 0 training examples: Analysis of ID3 mean = 48.077 (sd 9.615) versus PFOIL mean = 51.154 (sd 9.738). Difference: -3.077 Paired t-test results: t = -0.647, df = 9 (not significant for 1-tailed test) The 0.05 level confidence interval on the difference between the means: -11.794 to 5.640 For 10 training examples: Analysis of ID3 mean = 56.538 (sd 13.218) versus PFOIL mean = 68.077 (sd 10.885). Difference: -11.538* Paired t-test results: t = -2.620, df = 9 (significant at the 0.025 level for 1-tailed test) The 0.05 level confidence interval on the difference between the means: -19.611 to -3.466 For 25 training examples: Analysis of ID3 mean = 71.538 (sd 10.059) versus PFOIL mean = 74.615 (sd 9.911). Difference: -3.077 Paired t-test results: t = -0.605, df = 9 (not significant for 1-tailed test) The 0.05 level confidence interval on the difference between the means: -12.406 to 6.252 For 40 training examples: Analysis of ID3 mean = 71.154 (sd 6.934) versus PFOIL mean = 74.615 (sd 4.615). Difference: -3.462 Paired t-test results: t = -1.132, df = 9 (not significant for 1-tailed test) The 0.05 level confidence interval on the difference between the means: -9.067 to 2.144 For 80 training examples: Analysis of ID3 mean = 76.154 (sd 8.566) versus PFOIL mean = 77.308 (sd 5.000). Difference: -1.154 Paired t-test results: t = -0.461, df = 9 (not significant for 1-tailed test) The 0.05 level confidence interval on the difference between the means: -5.741 to 3.433 Comparing ID3 to KNN: For 0 training examples: Analysis of ID3 mean = 48.077 (sd 9.615) versus KNN mean = 49.615 (sd 9.948). Difference: -1.538 Paired t-test results: t = -0.361, df = 9 (not significant for 1-tailed test) The 0.05 level confidence interval on the difference between the means: -9.347 to 6.270 For 10 training examples: Analysis of ID3 mean = 56.538 (sd 13.218) versus KNN mean = 69.615 (sd 11.207). Difference: -13.077 Paired t-test results: t = -1.813, df = 9 (not significant for 1-tailed test) The 0.05 level confidence interval on the difference between the means: -26.296 to 0.142 For 25 training examples: Analysis of ID3 mean = 71.538 (sd 10.059) versus KNN mean = 75.769 (sd 12.047). Difference: -4.231 Paired t-test results: t = -0.661, df = 9 (not significant for 1-tailed test) The 0.05 level confidence interval on the difference between the means: -15.955 to 7.493 For 40 training examples: Analysis of ID3 mean = 71.154 (sd 6.934) versus KNN mean = 76.923 (sd 11.279). Difference: -5.769 Paired t-test results: t = -1.103, df = 9 (not significant for 1-tailed test) The 0.05 level confidence interval on the difference between the means: -15.358 to 3.820 For 80 training examples: Analysis of ID3 mean = 76.154 (sd 8.566) versus KNN mean = 79.615 (sd 7.892). Difference: -3.462 Paired t-test results: t = -0.906, df = 9 (not significant for 1-tailed test) The 0.05 level confidence interval on the difference between the means: -10.468 to 3.545 Comparing PFOIL to KNN: For 0 training examples: Analysis of PFOIL mean = 51.154 (sd 9.738) versus KNN mean = 49.615 (sd 9.948). Difference: 1.538 Paired t-test results: t = 0.332, df = 9 (not significant for 1-tailed test) The 0.05 level confidence interval on the difference between the means: -6.948 to 10.025 For 10 training examples: Analysis of PFOIL mean = 68.077 (sd 10.885) versus KNN mean = 69.615 (sd 11.207). Difference: -1.538 Paired t-test results: t = -0.252, df = 9 (not significant for 1-tailed test) The 0.05 level confidence interval on the difference between the means: -12.720 to 9.643 For 25 training examples: Analysis of PFOIL mean = 74.615 (sd 9.911) versus KNN mean = 75.769 (sd 12.047). Difference: -1.154 Paired t-test results: t = -0.203, df = 9 (not significant for 1-tailed test) The 0.05 level confidence interval on the difference between the means: -11.560 to 9.253 For 40 training examples: Analysis of PFOIL mean = 74.615 (sd 4.615) versus KNN mean = 76.923 (sd 11.279). Difference: -2.308 Paired t-test results: t = -0.502, df = 9 (not significant for 1-tailed test) The 0.05 level confidence interval on the difference between the means: -10.728 to 6.113 For 80 training examples: Analysis of PFOIL mean = 77.308 (sd 5.000) versus KNN mean = 79.615 (sd 7.892). Difference: -2.308 Paired t-test results: t = -0.643, df = 9 (not significant for 1-tailed test) The 0.05 level confidence interval on the difference between the means: -8.888 to 4.272 NIL ;;; If you don't care to save the actual training/test splits used and want to run a ;;; one-shot test, you can use RUN-STANDARD-TESTS instead. > (load "perceptron") ; Load the Perceptron neural-net learning system ;;; Loading binary file "perceptron.sbin" ;;; Loading binary file "/u/mooney/ml-progs/binary-encoder.sbin" #P"/tmp_mnt/v/sally/v1/mooney/ml-progs/perceptron.sbin" ;;; Run one-shot tests comparing ID3 and perceptron over 15 random trials ;;; with 80 max training examples, plotting points (10 25 40 80), using ;;; all remaining examples for testing, using "/u/mooney/data/dna-standard" ;;; as the data file, and "sample-dna-results2" to put the results in. > (run-standard-tests '(id3 perceptron) 15 80 '(10 25 40 80) nil "/u/mooney/data/dna-standard" "sample-dna-results2") ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 1: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 42.31% Training Time: 0.00 sec Test Time: 0.01 sec Concept Complexity: 1 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 38.46% Training Time: 0.81 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 0.29 sec Test Time: 0.01 sec Concept Complexity: 7 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 73.08% Training Time: 0.11 sec Test Time: 0.18 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 38.46% Training Time: 0.57 sec Test Time: 0.00 sec Concept Complexity: 10 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 84.62% Training Time: 0.43 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 0.82 sec Test Time: 0.01 sec Concept Complexity: 13 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 84.62% Training Time: 1.07 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 80.77% Training Time: 1.85 sec Test Time: 0.01 sec Concept Complexity: 31 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 88.46% Training Time: 2.05 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 2: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 50.00% Training Time: 0.00 sec Test Time: 0.00 sec Concept Complexity: 1 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 46.15% Training Time: 0.01 sec Test Time: 0.10 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 57.69% Training Time: 0.28 sec Test Time: 0.00 sec Concept Complexity: 7 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 84.62% Training Time: 0.14 sec Test Time: 0.14 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 84.62% Training Time: 0.77 sec Test Time: 0.00 sec Concept Complexity: 13 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 80.77% Training Time: 0.51 sec Test Time: 0.10 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 88.46% Training Time: 0.94 sec Test Time: 0.00 sec Concept Complexity: 16 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 92.31% Training Time: 0.81 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 65.38% Training Time: 1.79 sec Test Time: 0.00 sec Concept Complexity: 31 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 96.15% Training Time: 2.39 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 3: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 57.69% Training Time: 0.00 sec Test Time: 0.00 sec Concept Complexity: 1 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 61.54% Training Time: 0.01 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 50.00% Training Time: 0.18 sec Test Time: 0.01 sec Concept Complexity: 4 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 46.15% Training Time: 0.12 sec Test Time: 0.10 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 61.54% Training Time: 0.54 sec Test Time: 0.00 sec Concept Complexity: 10 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 80.77% Training Time: 0.47 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 53.85% Training Time: 1.05 sec Test Time: 0.00 sec Concept Complexity: 19 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 65.38% Training Time: 0.85 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 1.71 sec Test Time: 0.00 sec Concept Complexity: 28 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 2.48 sec Test Time: 0.10 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 4: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 46.15% Training Time: 0.00 sec Test Time: 0.00 sec Concept Complexity: 1 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 53.85% Training Time: 0.00 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 61.54% Training Time: 0.18 sec Test Time: 0.00 sec Concept Complexity: 4 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 69.23% Training Time: 0.10 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 0.45 sec Test Time: 0.01 sec Concept Complexity: 7 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 0.38 sec Test Time: 0.10 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 80.77% Training Time: 1.11 sec Test Time: 0.00 sec Concept Complexity: 19 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 73.08% Training Time: 0.88 sec Test Time: 0.16 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 73.08% Training Time: 1.82 sec Test Time: 0.01 sec Concept Complexity: 28 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 2.00 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 5: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 53.85% Training Time: 0.00 sec Test Time: 0.00 sec Concept Complexity: 1 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 50.00% Training Time: 0.00 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 0.18 sec Test Time: 0.01 sec Concept Complexity: 4 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 61.54% Training Time: 0.13 sec Test Time: 0.14 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 0.56 sec Test Time: 0.00 sec Concept Complexity: 10 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 69.23% Training Time: 0.55 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 69.23% Training Time: 0.80 sec Test Time: 0.01 sec Concept Complexity: 13 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 84.62% Training Time: 0.75 sec Test Time: 0.10 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 61.54% Training Time: 1.49 sec Test Time: 0.00 sec Concept Complexity: 22 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 84.62% Training Time: 1.99 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 6: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 53.85% Training Time: 0.00 sec Test Time: 0.00 sec Concept Complexity: 1 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 46.15% Training Time: 0.01 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 69.23% Training Time: 0.33 sec Test Time: 0.00 sec Concept Complexity: 7 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 69.23% Training Time: 0.12 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 53.85% Training Time: 0.67 sec Test Time: 0.00 sec Concept Complexity: 13 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 88.46% Training Time: 0.42 sec Test Time: 0.10 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 65.38% Training Time: 0.93 sec Test Time: 0.00 sec Concept Complexity: 16 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 88.46% Training Time: 0.91 sec Test Time: 0.10 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 88.46% Training Time: 1.40 sec Test Time: 0.00 sec Concept Complexity: 19 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 84.62% Training Time: 2.10 sec Test Time: 0.21 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 7: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 42.31% Training Time: 0.00 sec Test Time: 0.00 sec Concept Complexity: 1 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 50.00% Training Time: 0.00 sec Test Time: 0.10 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 38.46% Training Time: 0.27 sec Test Time: 0.00 sec Concept Complexity: 7 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 42.31% Training Time: 0.10 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 80.77% Training Time: 0.56 sec Test Time: 0.01 sec Concept Complexity: 10 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 80.77% Training Time: 0.42 sec Test Time: 0.15 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 84.62% Training Time: 0.77 sec Test Time: 0.00 sec Concept Complexity: 13 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 80.77% Training Time: 0.75 sec Test Time: 0.10 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 1.47 sec Test Time: 0.01 sec Concept Complexity: 22 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 88.46% Training Time: 2.09 sec Test Time: 0.10 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 8: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 38.46% Training Time: 0.00 sec Test Time: 0.00 sec Concept Complexity: 1 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 53.85% Training Time: 0.00 sec Test Time: 0.10 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 69.23% Training Time: 0.18 sec Test Time: 0.00 sec Concept Complexity: 4 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 65.38% Training Time: 0.13 sec Test Time: 0.13 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 0.54 sec Test Time: 0.01 sec Concept Complexity: 10 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 84.62% Training Time: 0.64 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 80.77% Training Time: 0.79 sec Test Time: 0.00 sec Concept Complexity: 13 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 92.31% Training Time: 0.86 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 73.08% Training Time: 1.81 sec Test Time: 0.00 sec Concept Complexity: 28 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 92.31% Training Time: 2.15 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 9: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 50.00% Training Time: 0.00 sec Test Time: 0.01 sec Concept Complexity: 1 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 50.00% Training Time: 0.01 sec Test Time: 0.12 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 80.77% Training Time: 0.20 sec Test Time: 0.01 sec Concept Complexity: 4 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 80.77% Training Time: 0.12 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 73.08% Training Time: 0.56 sec Test Time: 0.00 sec Concept Complexity: 10 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 84.62% Training Time: 0.42 sec Test Time: 0.10 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 84.62% Training Time: 0.67 sec Test Time: 0.01 sec Concept Complexity: 10 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 84.62% Training Time: 0.71 sec Test Time: 0.10 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 84.62% Training Time: 1.60 sec Test Time: 0.00 sec Concept Complexity: 25 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 84.62% Training Time: 1.71 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 10: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 42.31% Training Time: 0.00 sec Test Time: 0.00 sec Concept Complexity: 1 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 50.00% Training Time: 0.01 sec Test Time: 0.10 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 80.77% Training Time: 0.17 sec Test Time: 0.00 sec Concept Complexity: 4 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 61.54% Training Time: 0.09 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 65.38% Training Time: 0.45 sec Test Time: 0.00 sec Concept Complexity: 7 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 57.69% Training Time: 0.51 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 65.38% Training Time: 0.74 sec Test Time: 0.00 sec Concept Complexity: 13 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 88.46% Training Time: 0.85 sec Test Time: 0.10 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 61.54% Training Time: 1.84 sec Test Time: 0.00 sec Concept Complexity: 31 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 84.62% Training Time: 2.36 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 11: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 50.00% Training Time: 0.00 sec Test Time: 0.00 sec Concept Complexity: 1 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 50.00% Training Time: 0.01 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 61.54% Training Time: 0.35 sec Test Time: 0.00 sec Concept Complexity: 7 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 38.46% Training Time: 0.13 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 53.85% Training Time: 0.70 sec Test Time: 0.00 sec Concept Complexity: 13 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 88.46% Training Time: 0.45 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 65.38% Training Time: 0.93 sec Test Time: 0.00 sec Concept Complexity: 16 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 80.77% Training Time: 0.90 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 73.08% Training Time: 1.93 sec Test Time: 0.00 sec Concept Complexity: 31 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 84.62% Training Time: 2.09 sec Test Time: 0.21 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 12: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 38.46% Training Time: 0.00 sec Test Time: 0.01 sec Concept Complexity: 1 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 42.31% Training Time: 0.00 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 50.00% Training Time: 0.18 sec Test Time: 0.01 sec Concept Complexity: 4 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 50.00% Training Time: 0.12 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 69.23% Training Time: 0.44 sec Test Time: 0.00 sec Concept Complexity: 7 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 53.85% Training Time: 0.50 sec Test Time: 0.10 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 61.54% Training Time: 0.87 sec Test Time: 0.00 sec Concept Complexity: 16 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 65.38% Training Time: 0.76 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 69.23% Training Time: 1.59 sec Test Time: 0.01 sec Concept Complexity: 25 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 1.93 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 13: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 50.00% Training Time: 0.00 sec Test Time: 0.01 sec Concept Complexity: 1 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 23.08% Training Time: 0.00 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 65.38% Training Time: 0.27 sec Test Time: 0.00 sec Concept Complexity: 7 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 61.54% Training Time: 0.15 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 57.69% Training Time: 0.68 sec Test Time: 0.00 sec Concept Complexity: 13 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 0.47 sec Test Time: 0.10 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 69.23% Training Time: 0.82 sec Test Time: 0.01 sec Concept Complexity: 13 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 73.08% Training Time: 0.84 sec Test Time: 0.16 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 69.23% Training Time: 1.71 sec Test Time: 0.00 sec Concept Complexity: 28 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 88.46% Training Time: 1.90 sec Test Time: 0.10 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 14: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 42.31% Training Time: 0.00 sec Test Time: 0.00 sec Concept Complexity: 1 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 46.15% Training Time: 0.01 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 46.15% Training Time: 0.18 sec Test Time: 0.01 sec Concept Complexity: 4 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 53.85% Training Time: 0.12 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 57.69% Training Time: 0.57 sec Test Time: 0.00 sec Concept Complexity: 10 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 65.38% Training Time: 0.45 sec Test Time: 0.14 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 0.64 sec Test Time: 0.00 sec Concept Complexity: 10 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 76.92% Training Time: 0.89 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 73.08% Training Time: 1.61 sec Test Time: 0.00 sec Concept Complexity: 25 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 92.31% Training Time: 2.59 sec Test Time: 0.10 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Results for trial 15: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 0 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 50.00% Training Time: 0.00 sec Test Time: 0.00 sec Concept Complexity: 1 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 57.69% Training Time: 0.00 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 10 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 53.85% Training Time: 0.30 sec Test Time: 0.00 sec Concept Complexity: 7 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 69.23% Training Time: 0.12 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 25 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 84.62% Training Time: 0.65 sec Test Time: 0.01 sec Concept Complexity: 13 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 80.77% Training Time: 0.56 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 40 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 73.08% Training Time: 0.82 sec Test Time: 0.00 sec Concept Complexity: 13 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 80.77% Training Time: 0.99 sec Test Time: 0.10 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test results for 80 training examples: Training ID3...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 69.23% Training Time: 1.70 sec Test Time: 0.00 sec Concept Complexity: 28 Training PERCEPTRON...Testing Training Data...Testing Test Data... Training Accuracy: 100.00% Test Accuracy: 92.31% Training Time: 1.91 sec Test Time: 0.11 sec Concept Complexity: NIL ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ NIL ;;; Run a t-test to compare the new test accuracy results > (t-test-file "sample-dna-results2") Total number of samples in data: 15 Comparing ID3 to PERCEPTRON: For 0 training examples: Analysis of ID3 mean = 47.179 (sd 5.699) versus PERCEPTRON mean = 47.949 (sd 8.642). Difference: -0.769 Paired t-test results: t = -0.305, df = 14 (not significant for 2-tailed test) The 0.05 level confidence interval on the difference between the means: -6.182 to 4.644 For 10 training examples: Analysis of ID3 mean = 62.564 (sd 12.754) versus PERCEPTRON mean = 61.795 (sd 13.059). Difference: 0.769 Paired t-test results: t = 0.233, df = 14 (not significant for 2-tailed test) The 0.05 level confidence interval on the difference between the means: -6.305 to 7.843 For 25 training examples: Analysis of ID3 mean = 67.436 (sd 12.938) versus PERCEPTRON mean = 76.923 (sd 10.320). Difference: -9.487 Paired t-test results: t = -2.038, df = 14 (not significant for 2-tailed test) The 0.05 level confidence interval on the difference between the means: -19.474 to 0.500 For 40 training examples: Analysis of ID3 mean = 73.077 (sd 9.525) versus PERCEPTRON mean = 80.769 (sd 8.309). Difference: -7.692* Paired t-test results: t = -3.282, df = 14 (significant at the 0.01 level for 2-tailed test) The 0.05 level confidence interval on the difference between the means: -12.720 to -2.664 For 80 training examples: Analysis of ID3 mean = 73.077 (sd 7.431) versus PERCEPTRON mean = 86.154 (sd 5.756). Difference: -13.077* Paired t-test results: t = -4.888, df = 14 (significant at the 0.001 level for 2-tailed test) The 0.05 level confidence interval on the difference between the means: -18.815 to -7.339 NIL ;;; Check the training accuracy > (t-test-file "sample-dna-results2" 2 'train-accuracy) Total number of samples in data: 15 Comparing ID3 to PERCEPTRON: For 0 training examples: Analysis of ID3 mean = 100.000 (sd 0.000) versus PERCEPTRON mean = 100.000 (sd 0.000). Difference: 0.000 Paired t-test results: t = 0.000, df = 14 (not significant for 2-tailed test) The 0.05 level confidence interval on the difference between the means: 0.000 to 0.000 For 10 training examples: Analysis of ID3 mean = 100.000 (sd 0.000) versus PERCEPTRON mean = 100.000 (sd 0.000). Difference: 0.000 Paired t-test results: t = 0.000, df = 14 (not significant for 2-tailed test) The 0.05 level confidence interval on the difference between the means: 0.000 to 0.000 For 25 training examples: Analysis of ID3 mean = 100.000 (sd 0.000) versus PERCEPTRON mean = 100.000 (sd 0.000). Difference: 0.000 Paired t-test results: t = 0.000, df = 14 (not significant for 2-tailed test) The 0.05 level confidence interval on the difference between the means: 0.000 to 0.000 For 40 training examples: Analysis of ID3 mean = 100.000 (sd 0.000) versus PERCEPTRON mean = 100.000 (sd 0.000). Difference: 0.000 Paired t-test results: t = 0.000, df = 14 (not significant for 2-tailed test) The 0.05 level confidence interval on the difference between the means: 0.000 to 0.000 For 80 training examples: Analysis of ID3 mean = 100.000 (sd 0.000) versus PERCEPTRON mean = 100.000 (sd 0.000). Difference: 0.000 Paired t-test results: t = 0.000, df = 14 (not significant for 2-tailed test) The 0.05 level confidence interval on the difference between the means: 0.000 to 0.000 NIL ;;; Also compare training time. > (t-test-file "sample-dna-results2" 2 'train-time) Total number of samples in data: 15 Comparing ID3 to PERCEPTRON: For 0 training examples: Analysis of ID3 mean = 0.000 (sd 0.000) versus PERCEPTRON mean = 0.059 (sd 0.201). Difference: -0.059 Paired t-test results: t = -1.093, df = 14 (not significant for 2-tailed test) The 0.05 level confidence interval on the difference between the means: -0.174 to 0.056 For 10 training examples: Analysis of ID3 mean = 0.236 (sd 0.062) versus PERCEPTRON mean = 0.120 (sd 0.015). Difference: 0.116* Paired t-test results: t = 7.277, df = 14 (significant at the 0.001 level for 2-tailed test) The 0.05 level confidence interval on the difference between the means: 0.082 to 0.150 For 25 training examples: Analysis of ID3 mean = 0.581 (sd 0.094) versus PERCEPTRON mean = 0.479 (sd 0.066). Difference: 0.102* Paired t-test results: t = 3.400, df = 14 (significant at the 0.01 level for 2-tailed test) The 0.05 level confidence interval on the difference between the means: 0.038 to 0.166 For 40 training examples: Analysis of ID3 mean = 0.847 (sd 0.125) versus PERCEPTRON mean = 0.855 (sd 0.092). Difference: -0.008 Paired t-test results: t = -0.212, df = 14 (not significant for 2-tailed test) The 0.05 level confidence interval on the difference between the means: -0.089 to 0.073 For 80 training examples: Analysis of ID3 mean = 1.688 (sd 0.152) versus PERCEPTRON mean = 2.116 (sd 0.233). Difference: -0.428* Paired t-test results: t = -6.259, df = 14 (significant at the 0.001 level for 2-tailed test) The 0.05 level confidence interval on the difference between the means: -0.575 to -0.281 NIL