Finding Conditional Mutual Information From 3 Discrete Variable

I am trying to find conditional mutual information between three discrete random variable using pyitlib package for python with the help of the formula: I(X;Y|Z)=H(X|Z)+H(Y|Z)-H(X,

Solution 1:

Based on the definitions of conditional entropy, calculating in bits (i.e. base 2) I obtain H(X|Z)=0.784159, H(Y|Z)=0.325011, H(X,Y|Z) = 0.950826. Based on the definition of conditional mutual information you provide above, I obtain I(X;Y|Z)=H(X|Z)+H(Y|Z)-H(X,Y|Z)= 0.158344. Noting that pyitlib uses base 2 by default, drv.information_mutual_conditional(X,Y,Z) appears to be computing the correct result.

Note that your use of drv.entropy_conditional(X,Y,Z) in your first example to compute conditional entropy is incorrect, you can however use drv.entropy_conditional(XY,Z), where XY is a 1D array representing the joint observations about X and Y, for example XY = [2*xy[0] + xy[1] for xy in zip(X,Y)].

Solution 2:

I think that the library function entropy_conditional(x,y,z) has some errors. I also test my samples, the same problem happens. however, the function entropy_conditional with two variables is ok. So I code my entropy_conditional(x,y,z) as entropy(x,y,z), the results is correct. the code may be not beautiful.

def gen_dict(x):
    dict_z = {}
    for key in x:
        dict_z[key] = dict_z.get(key, 0) + 1
    return dict_z

def entropy(x,y,z):   
    x = np.array([x,y,z]).T
    x = x[x[:,-1].argsort()] # sorted by the last column    
    w = x[:,-3]
    y = x[:,-2]
    z = x[:,-1]
    
    # dict_w = gen_dict(w)
    # dict_y = gen_dict(y)
    dict_z = gen_dict(z)
    list_z = [dict_z[i] for i in set(z)]
    p_z = np.array(list_z)/sum(list_z)
    pos = 0
    ent = 0
    for i in range(len(list_z)):   
        w = x[pos:pos+list_z[i],-3]
        y = x[pos:pos+list_z[i],-2]
        z = x[pos:pos+list_z[i],-1]
        pos += list_z[i]
        list_wy = np.zeros((len(set(w)),len(set(y))), dtype = float , order ="C")
        list_w = list(set(w))
        list_y = list(set(y))
        
        for j in range(len(w)):
            pos_w = list_w.index(w[j])
            pos_y = list_y.index(y[j])
            list_wy[pos_w,pos_y] += 1
            #print(pos_w)
            #print(pos_y)
        list_p = list_wy.flatten()
        list_p = np.array([k for k in list_p if k>0]/sum(list_p))
        ent_t = 0
        for j in list_p:
            ent_t += -j * math.log2(j)
        #print(ent_t)
        ent += p_z[i]* ent_t
    return ent
    

X=[0,1,1,0,1,0,1,0,0,1,0,0]
Y=[0,1,1,0,0,0,1,0,0,1,1,0]
Z=[1,0,0,1,1,0,0,1,1,0,0,1]  

a=drv.entropy_conditional(X,Z)
##print(a)
b=drv.entropy_conditional(Y,Z)         
c = entropy(X, Y, Z)
p=a+b-c
print(p)
0.15834454415751043