The availability of genome-wide expression data to complement the measurements of a phenotypic trait opens new opportunities for identifying biologic processes and genes that are involved in trait expression. Usually differential analysis is a preliminary step to identify the key biological processes involved in the variability of the trait of interest. However, this variability shall be viewed as resulting from a complex combination of genes individual contributions. In other words, exploring the interactions between genes viewed in a network structure which vertices are genes and edges stand for inhibition or activation connections gives much more insight on the internal structure of expression profiles. Many currently available solutions for network analysis based on the Gaussian Graphical Model have been developed but an efficient estimation of the network from high-dimensional data is still a questioning issue. Extending the idea introduced for differential analysis by Friguet et al. 2009 and Blum et al. 2010, we propose to take advantage of a factor model structure to infer gene regulatory networks. This method shows good inferential properties and also allows an efficient testing strategy for the significance of direct gene interactions (partial correlations). We use the method in a study that aims at identifying the genes implied in fatness variability in chickens. We model the networks of genes controlled by genome regions known to be related to the fatness variability and analyze the modular structure thanks to the annotation available giving us new hypotheses about the causal mutations and underlying biological processes.