split a big data set into several subsets

This questions comes from when I want to email a dataset, it's too large to email. So I need to split it into several subsets to email.

Usually for two variables data, 1 million obs will be about 10MB after gzip. This is a proper size to email.

Here shows an example how to split a 10000 records data into several subdata with each having about 2300 obs.

options mprint mlogic;

data test;

  do i=1 to 10000;

    x=rannor(1);

      output;

  end;

run;

%let n_obs=2300;

proc sql noprint;

  select count(1) into :m_total from test;

quit;

%macro sub_data;

    %let n_dataset=%sysfunc(ceil((&m_total/&n_obs)));

      %do i=1 %to &n_dataset;

        data sub_data_&i;

          set test;

        if (&i-1)*&n_obs+1<=_n_<=&i*&n_obs;

        run;

      %end;

%mend sub_data;

%sub_data;

how to sample several groups of data from a given data set, sampled data will not appear again?

e.g., data test has 10000 obs, we want to sample 3 data sets, called sample_1, sample_2 and sample_3. The obs in sample_1 will not appear in sample_2, the obs in sample 1, sample_2 will not appear in sample_3.

In the following we use an indicator called group. For sample_1, we srs from data test, and id its group=1. For sample_2, and so on, we restrict group<1 to exclude data in sample_1. And id sample_2 as group=2, and so on.

options mprint mlogic;

data test;

  do i=1 to 10000;

    x=rannor(1);

      output;

  end;

run;

%let n_sample=3;

%macro m_sample;

  %do i=1 %to &n_sample;

    proc surveyselect data=test %if &i>1 %then %do; (where=(group<1)) %end; method=srs sampsize=100 out=sample_&i;

    run;

      proc sort data=sample_&i;

        by i;

      run;

      proc sort data=test;

        by i;

      run;

      data test;

        merge test sample_&i(in=in2);

        by i;

        if in2 then group=&i;

      run;

  %end;

%mend m_sample;

%m_sample;

gzip several different files in linux

e.g., during sas running, there are several output and we want to send out these different output in one email.

1: use proc printto to print to lst or print to csv files; if for lst, remember to cat xx.lst >> xx.txt for easy to read.

2: tar different files together.   x "tar -cvf  to_be_together.tar  file1.csv file2.txt file3.log"; This step to tar file1, file2, file3 together into one file called to_be_together.tar

3: x "gzip -f to_be_together.tar"

that's it.

zz: A question about how to use _N_

The original is from mysas(probably? Forgot it). Look at the answer and pay attention how to use _N_.
 

Question：

data a;

input id $ x;

cards;

a 0

a 1

a 2

a 1

b 3

c 0

c 3

d 2

d 2

e 1

;

run;

* using the data above, how to get a variable x_grp, which is made of distinct value of x in each id group. The data should be like:

id x x_grp

a 0 0_1_2

a 1 0_1_2

a 2 0_1_2

a 1 0_1_2

b 3 3

c 0 0_3

c 3 0_3

d 2 2

d 2 2

e 1 1

Answers：

Method 1：

data b;

    length x_grp $50;

    do _n_=1 by 1 until(last.id);

        set a;

        by id notsorted;

        if indexw(trim(x_grp),cats(x),'_')=0 then x_grp=catx('_',x_grp,x);

    end;

    do _n_=1 to _n_;

        set a;

        output;

    end;

run;

Method 2：

data a2;

set a;

retain grp;

by id notsorted;

if first.id then grp=x;

else if find(grp,compress(x))=0 then grp=catx('_',grp,x);

if last.id then output;

run;

data b;

merge a a2;

by id;

run;

proc freq v.s. proc sql

sasuser.sale2000 has variable origin and dest, if we want to list the unique combination of these two variables, we can use proc sql; select distinct ... Another method is using proc freq. It not only gives the no duplicated combination, but also sorts the result by the tables variables. The sas code is:

options formdlim='-' ;

proc sql;

    create table sqlrst as

    select distinct origin, dest

      from sasuser.sale2000

      order by origin, dest;

quit;

proc print data=sqlrst width=min;

run;

proc freq data=sasuser.sale2000 noprint;

    tables origin*dest / list out=freq1;

run;

proc print data=freq1(drop=count percent) width=min;

run;