***** To join INSNA, visit http://www.insna.org *****
On 13 Jul 2005, at 11:11, elijah wright wrote:
> ***** To join INSNA, visit http://www.insna.org *****
> the size of the dense matrix would be n(n-1)/2, which for a 1M node
> means that you would have 500+GB of data, plus overhead. probably a
> couple of terabytes of data, in the end. expensive!
> storing the raw matrix isn't typically done with super-huge
> networks. it
> is horribly wasteful when networks are often actually quite
> sparse... you
> want to store the sparse nets.
> look for things related to the Harwell-Boeing format. pretty common.
> or just store lists of nodes and edges, and query them out of your
> store as you need them. think about what your research questions
> are, and
> store the data in ways that you can query to answer those questions.
> when people start talking about million-node networks, we usually
> talking about sampling and modeling. you CAN NOT process a network of
> that size with available hardware. it just isn't going to happen.
> very small (this is always relative, of course), sampled networks will
> chew up all of the resources available on a high-end desktop machine.
I have processed networks of that size (e.g., a tree of 700K
vertices, and a graph of 300K with 2.2M edges) using JUNG. I don't
visualize them, and I don't do analyses on them that run in greater
than O(m) time (m: # of edges), but that leaves quite a lot of
scope. This includes some metadata for each vertex and edge, and
occasionally some extraction of subgraphs; I can do this on a machine
with 1.5 GB of Java heap space, and could do larger networks if I
wanted to put more RAM in the machine (or leave less overhead for
> sorry if this seems a little low-level. i'd be happy to talk off-list
> about things that we've tried and that we've found helpful or not --
> particularly with regard to very large social networks.
I'd be interested to hear what you've done in this area.
firstname.lastname@example.org...Obscurium Per Obscurius...www.ics.uci.edu/~jmadden
Joshua O'Madadhain: Information Scientist, Musician, and Philosopher-
It's that moment of dawning comprehension that I live for--Bill
My opinions are too rational and insightful to be those of any
SOCNET is a service of INSNA, the professional association for social
network researchers (http://www.insna.org). To unsubscribe, send
an email message to [log in to unmask] containing the line
UNSUBSCRIBE SOCNET in the body of the message.