*****  To join INSNA, visit  *****

On 25 Jan 2006, at 14:39, Andrew Cleary wrote:

> It is my suggestion to be pretty skeptical of the ability of these  
> types of programs, not because the *will* isn't there (it is,  
> though I sure wish it wasn't), but because these are very difficult  
> technical problems to pull off. Having access to a lot of data  
> doesn't wash away the problems of telling the useful data from  
> noise. Just because someone hangs a scary government three-letter- 
> acronym outside their door doesn't mean they have magic bullets  
> that give them solutions to longstanding, very difficult problems,  
> and the incentive structures in place in government labs are not  
> very good ones for encouraging innovative solutions to problems  
> that the rest of the world hasn't solved. For example, they say  
> below that the government's systems "connect the dots", but  
> frankly, those "dots" are subject to the same problems of  
> canonicalization, the need for ontologies to apply *semantics* to  
> that data, data trustworthiness and cleansing, natural language  
> processing, inference in small and noisy datasetc, etc., etc., that  
> very small data mining projects have, but at a *much* larger scale,  
> and *all* of those issues need to be solved for the whole system to  
> really start making sense out of anything.
> IOW: it would be my estimate that these kinds of programs are much  
> more bark than bite.


You are, of course, correct in stating that there are a lot of  
problems to be solved here.  However, in my opinion, there are  
nevertheless reasons to be concerned about "these types of programs".

(1) NSA (and other "scary government three-letter-acronym"  
agencies :) ) employ a lot of very smart people that have access to  
large amounts of resources, are aware of these problems (it's  
probably safe to say that they read this list, for one thing), and  
have been working with large data sets for some time.  I don't  
necessarily assume that they have secret methods that blow the doors  
off anything known to the academic or industrial world, but I would  
not underestimate the amount of effort that they're willing to put  
into addressing these problems.  Also, of course, it's worth  
mentioning that even if you're right about their attitude towards  
innovation, they have all the public research in this area to draw on.

(2) That said, the potential for misuse--whether politically  
motivated or not--of such programs is immense.  Ironically, one of  
the most likely scenarios, in my opinion, for misuse is one in which  
the models don't work very well, but are used anyway...and end up  
connecting all sorts of dots that should not be connected.  I suspect  
that we've actually been seeing evidence of this already.

Joshua O'Madadhain

> Cheers,
> Andy, formerly of DOE research labs...
> Dr Andrew J Cleary
> Director of Algorithms R&D
> Visible Path Corporation
> [log in to unmask]
> -----Original Message-----
> From: Social Networks Discussion Forum on behalf of Barry Wellman
> Sent: Wed 1/25/2006 2:04 PM
> To: [log in to unmask]
> Subject: networked spyware
> *****  To join INSNA, visit  *****
> NSA spy program hinges on state-of-the-art technology
> By Shane Harris, National Journal
> The furor over the National Security Agency's domestic eavesdropping,
> authorized by President Bush, has focused largely on legal  
> questions --
> whether the NSA has the authority to spy on Americans inside the  
> United
> States and whether the commander-in-chief can order the agency to  
> do so.
> But that debate has largely smothered examination of how the nation's
> largest intelligence agency is collecting -- and analyzing --  
> information
> intercepted from hundreds, possibly thousands, of Americans. Since the
> 9/11 attacks, the NSA has abandoned the mantra that guided it in  
> earlier
> decades -- Do not spy on Americans inside the nation's borders. Things
> have changed, and the NSA may be on the cusp of employing state-of- 
> the-art
> technologies to uncover more information about potential  
> terrorists, and
> about Americans here at home.
> ....
> the NSA has pursued cutting-edge data-mining technologies that  
> don't just
> find key words but also uncover hidden relationships among data  
> points.
> These technologies can even detect how a particular analyst thinks,
> identify what his or her biases are, and then suggest alternative
> hypotheses.
> Data-mining systems, which the NSA has publicly pursued and spent  
> millions
> of dollars researching, don't just "connect the dots" but also alert
> analysts about which dots to connect, which to disregard, and how to
> connect them in ways they may never have considered. It is unclear  
> which,
> if any, of these data-mining tools the NSA is using to analyze the
> domestic information gathered in the current eavesdropping program,  
> but
> the tools themselves offer a telling look into the agency's  
> potential to
> exploit what it collects, regardless of its legal basis for doing so.
>  etc
>  Barry
>  _____________________________________________________________________ Per
Joshua O'Madadhain: Information Scientist, Musician, and Philosopher- 
   It's that moment of dawning comprehension that I live for--Bill  
   My opinions are too rational and insightful to be those of any  

SOCNET is a service of INSNA, the professional association for social
network researchers ( To unsubscribe, send
an email message to [log in to unmask] containing the line
UNSUBSCRIBE SOCNET in the body of the message.