;; Paritosh, 02/8/04 23:49:56

;; This is a proof of concept demonstration of CARVE for Cog Sci 2004
;; Many steps here are tedious and all should be automated, and nothing
;; done in Matlab. 

;; Only African countries were involved. All 54 countries of the African
;; continent. 

;; First we create the mapping between the names used by Cyc and KSL KBs. 
;; This information is in wfb-africa-extractor.lsp in the variable 
;; *wfb-cyc-name-mappings*. This was done partly by hand. 

;; Then we go into the WFB, and extract all facts for each of these 54 
;; countries using gather-wfb-facts-for-country, and this information is
;; saved in africa_wfb.lsp in the variable *wfb-africa-cache* which is a
;; list of entries like (<country> <list of facts>) -- these are all the 
;; facts for the country in WFB. This is done to speed things up. 

;; Now, on an average there are 200+ facts in the WFB for each of the 
;; countries, and we dont know how to translate them into CYC compatible
;; representation automatically. Right now I am interested in numbers. 

;; So we extract 10 interesting numeric dimensions using gather-selected-facts
;; defined in ksl-to-cyc-v2.lsp.

;; And this information is saved in africa_wfb_selected.lsp in the 
;; variable *wfb-africa-selected-facts* in the same format as in the total 
;; cache described above. 

;; The next step is to cluster these numeric dimensions to obtain the 
;; first cut dimensional partitions. For now we chose to use k-means 
;; algorithm for this clustering and produce three clusters. For doing 
;; this, we need to take the data to matlab. My matlab script 
;; cluster_all_dimensions.m which calls cluster_analysis.m will do 
;; this job, but before that, we need to save the data as a matrix. 

;; The function convert-data-to-array does that and saves a text file 
;; with row and column labels in africa_wfb_selected.txt. This is then 
;; loaded in Excel, and I manually made sure that there were no nils 
;; (there were some since imports/exports for Eritrea were missing for
;; example), and replaced absent values by NaNs for Matlab. 

;; Then only the numeric part of the matrix (sans row/col labels) was saved as
;; africa_wfb_selected_data_for_matlab.txt. This is read in matlab
;; as AFR[][]

;; Now on to MATLAB. 


>> afr=load('africa_wfb_selected_data_for_matlab.txt');
>> size(afr)
ans =
    54    10

;; What do we need here as output from Matlab? 
;; For every dimension, we want three clusters. And we want to produce a 
;; matrix C, in which C[i,j] refers to the ith country and jth field in the
;; data file above, and is 0,1,or 2, depending upon if the value of the
;; AFR[i,j] belongs to the lo, med or hi cluster. 

;; cluster analysis done

;; africa_clusters.txt/xls/lsp contains the results of k-means clustering. 
;; Now how do we get it into symbolic representations in the KB? 

;; High/Medium/LowValueFn [Cyc calls them AmountFn, and we need a new
;; name, value is more general and works]

;; (HighValueFn  birthRate <context>)
;; What is the context? The set of examples?

;; Notes from meeting with Ken regarding representation. We should have
;; three kinds of functions that we will add for quantities (in addition
;; to Cyc's High/Medium/Low